Discussion:
Internals and Data Structures - DLM information
(too old to reply)
David Froble
2017-01-05 06:29:20 UTC
Permalink
I'm looking at a half dozen books on Amazon, all VMS internals and such
publications. What I'm looking for is information on the DLM, data structures
and design.

There is a V5.2 from 1987. Not sure if this is what I'm looking for. Can
anyone with the books check in theirs and tell me which has DLM information.

Dave
V***@SendSpamHere.ORG
2017-01-05 13:45:41 UTC
Permalink
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and such
publications. What I'm looking for is information on the DLM, data structures
and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking for. Can
anyone with the books check in theirs and tell me which has DLM information.
VAX/VMS Internals and Data Structures -- Version 5.2
Ruth Goldenberg/Order Number EY-C171E-DP/ISBN 1-55558-059-9

Part III/Synchronization
Chapter 10 --- Lock Management
10.1 - Lock Management Data Structures
10.2 - Lock Management System Services
10.3 - Handling Deadlocks
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
Stephen Hoffman
2017-01-05 14:50:31 UTC
Permalink
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and such
publications. What I'm looking for is information on the DLM, data
structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking for.
Can anyone with the books check in theirs and tell me which has DLM
information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...

Beyond the two manuals in the doc set and articles in the DEC and VMS
Technical Journals, Roy Davis' VAXcluster Principles book, most likely.
ISBN was 55558-112-9

Less so the IDSM. The DLM gets a fairly general discussion in the
IDSM, but not much in the way of details.

Or the OpenVMS source listings, for the full details.

Or for more general interest, skim through the OpenSSL open-source
variant of clustering software that Compaq released; that was a Unix
DLM similar to the OpenVMS DLM. Or skim the books and resources about
one of the more recent variants; see
https://en.wikipedia.org/wiki/Distributed_lock_manager for some
pointers to those.

If you do want to go for an IDSM for this or other reasons, you'll be
looking at a mix of V6 and the V7 updates to get all of the information
with the "latest" available information. The V6 stuff was the last
full copy of the IDSM, the V7 subset documents added chapters with some
details on specific and then-newer areas.

Closely related to the DLM is the distributed transaction manager;
DECdtm, and also the DEC RTR and the more recent open-source
0MQ/ActiveMQ/RabbitMQ messaging packages.
--
Pure Personal Opinion | HoffmanLabs LLC
Stephen Hoffman
2017-01-05 15:14:34 UTC
Permalink
...The DLM gets a fairly general {design and usage} discussion in the
IDSM, but not much in the way of {design and usage} details.
It's... Internals and Data Structures. But if that's what you're
looking for here...
--
Pure Personal Opinion | HoffmanLabs LLC
David Froble
2017-01-05 17:33:46 UTC
Permalink
Post by Stephen Hoffman
...The DLM gets a fairly general {design and usage} discussion in the
IDSM, but not much in the way of {design and usage} details.
It's... Internals and Data Structures. But if that's what you're
looking for here...
I've used the DLM quite a bit. Usage isn't a problem. But now I'm looking at
internal design, and I need some documentation.
David Froble
2017-01-05 15:23:00 UTC
Permalink
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking for.
Can anyone with the books check in theirs and tell me which has DLM
information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.

I just don't have enough information now, and, will need some help, as right
now, I'd come up with a solution looking for a problem. For instance, Postgre
database, how large a range is reasonable, and if talking bytes, how many should
be supported, which would determine the size of integer required. I figure a
database could be rather large.

Don't know that I need exact information on the data structures. Whoever
implements such a modification would have to work from what's current in VMS.
But I need enough information to get close to what might be required.

One thing seems to be clear to me, it's not just the DLM, but I/O stuff would
have to also be implemented to use the new feature. There it gets interesting.
If using I/O buffers, how to determine the size for one thing. Or perhaps
direct I/O into the "on disk" file?
Post by Stephen Hoffman
Beyond the two manuals in the doc set and articles in the DEC and VMS
Technical Journals, Roy Davis' VAXcluster Principles book, most likely.
ISBN was 55558-112-9
Less so the IDSM. The DLM gets a fairly general discussion in the IDSM,
but not much in the way of details.
Or the OpenVMS source listings, for the full details.
Or for more general interest, skim through the OpenSSL open-source
variant of clustering software that Compaq released; that was a Unix DLM
similar to the OpenVMS DLM. Or skim the books and resources about one
of the more recent variants; see
https://en.wikipedia.org/wiki/Distributed_lock_manager for some pointers
to those.
If you do want to go for an IDSM for this or other reasons, you'll be
looking at a mix of V6 and the V7 updates to get all of the information
with the "latest" available information. The V6 stuff was the last
full copy of the IDSM, the V7 subset documents added chapters with some
details on specific and then-newer areas.
Closely related to the DLM is the distributed transaction manager;
DECdtm, and also the DEC RTR and the more recent open-source
0MQ/ActiveMQ/RabbitMQ messaging packages.
Stephen Hoffman
2017-01-05 15:59:59 UTC
Permalink
Post by David Froble
Briefly, I'm looking at the issue of byte range locking. My
intentions, if I'm able to do so, are to come up with a preliminary
design for additions to the VMS DLM to implement byte range locking,
then distribute it to those interested to comment upon, add comments
and suggestions, and present the result to VSI as a potential
modification. It would be up to VSI as to whether to include it in VMS.
There's nothing on the byte-range locking implementation in the IDSM or
VAXcluster books. That implementation is entirely too new, and — other
than the C API — largely undocumented. IIRC, somebody did some
reverse-engineering and figured out parts, but there's been little
published on the topic from HP/HPE/VSI.
Post by David Froble
I just don't have enough information now, and, will need some help, as
right now, I'd come up with a solution looking for a problem. For
instance, Postgre database, how large a range is reasonable, and if
talking bytes, how many should be supported, which would determine the
size of integer required. I figure a database could be rather large.
Don't know that I need exact information on the data structures.
Whoever implements such a modification would have to work from what's
current in VMS. But I need enough information to get close to what
might be required.
One thing seems to be clear to me, it's not just the DLM, but I/O stuff
would have to also be implemented to use the new feature. There it
gets interesting. If using I/O buffers, how to determine the size for
one thing. Or perhaps direct I/O into the "on disk" file?
Not sure how you're envisioning I/O would be connected to the DLM.
Something else will usually map storage ranges and caches, and that
mapping and caching might or will be coordinated via DLM, but the I/O
itself is likely going to be isolated from the DLM. In OpenVMS, RMS
coordinates the caches in a cluster, if that's where you're headed
here. There's discussion of that sort of coordination in Davis' book.

If you're thinking of porting and using PostgreSQL on OpenVMS, then the
OpenVMS problems with shared stream I/O (SSIO) and a few other details
will arise. You've been involved in that discussion before, though.
For those that haven't been, there's an overview of SSIO (for
PostgreSQL or otherwise) at
http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf and
the discussions of SSIO-related bugs and problems are buried in the
OpenVMS bug lists, and probably a few postings around the 'net.

If you're using the DLM for your own resource coordination outside of C
(and not of something like PostgreSQL and its locking and I/O
activities via the C library), then you get to roll your own byte-range
locking if that's what you need. AFAIK, what OpenVMS C is doing there
for byte-range locking is specific to C applications, too. File and
record locks are general.

As for application design, I've had success with just mapping the
entire file. That's not a typical OpenVMS design, but it does work
nicely, and it makes some tasks simpler. Reliability and crash
recovery from journals and/or from a shadowed server. Or using code to
unmarshall the data into memory, and then operating there, and
marshalling the data back out for a clean shutdown or such. Again,
with a journal written to storage or to another host or such, for
purposes of crash recovery.

Some general locking information, and not specific to OpenVMS and the
DLM: http://apenwarr.ca/log/?m=201012#13
--
Pure Personal Opinion | HoffmanLabs LLC
David Froble
2017-01-05 17:45:48 UTC
Permalink
Post by Stephen Hoffman
Post by David Froble
Briefly, I'm looking at the issue of byte range locking. My
intentions, if I'm able to do so, are to come up with a preliminary
design for additions to the VMS DLM to implement byte range locking,
then distribute it to those interested to comment upon, add comments
and suggestions, and present the result to VSI as a potential
modification. It would be up to VSI as to whether to include it in VMS.
There's nothing on the byte-range locking implementation in the IDSM or
VAXcluster books. That implementation is entirely too new, and — other
than the C API — largely undocumented. IIRC, somebody did some
reverse-engineering and figured out parts, but there's been little
published on the topic from HP/HPE/VSI.
Post by David Froble
I just don't have enough information now, and, will need some help, as
right now, I'd come up with a solution looking for a problem. For
instance, Postgre database, how large a range is reasonable, and if
talking bytes, how many should be supported, which would determine the
size of integer required. I figure a database could be rather large.
Don't know that I need exact information on the data structures.
Whoever implements such a modification would have to work from what's
current in VMS. But I need enough information to get close to what
might be required.
One thing seems to be clear to me, it's not just the DLM, but I/O
stuff would have to also be implemented to use the new feature. There
it gets interesting. If using I/O buffers, how to determine the size
for one thing. Or perhaps direct I/O into the "on disk" file?
Not sure how you're envisioning I/O would be connected to the DLM.
I'm not. The DLM should be sort of an object that does a job. However, there
is nothing that I'm aware of that does byte range locking, or has that need,
that's currently implemented. Perhaps I should ignore the I/O part of it and
just concentrate on doing locks on byte ranges.
Post by Stephen Hoffman
Something else will usually map storage ranges and caches, and that
mapping and caching might or will be coordinated via DLM, but the I/O
itself is likely going to be isolated from the DLM. In OpenVMS, RMS
coordinates the caches in a cluster, if that's where you're headed
here. There's discussion of that sort of coordination in Davis' book.
If you're thinking of porting and using PostgreSQL on OpenVMS,
Slow down Steve, that's beyond me. What I'm looking at is a capability that
those wishing to do such a port say they need.
Post by Stephen Hoffman
then the
OpenVMS problems with shared stream I/O (SSIO) and a few other details
will arise. You've been involved in that discussion before, though.
For those that haven't been, there's an overview of SSIO (for PostgreSQL
or otherwise) at
http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf and
the discussions of SSIO-related bugs and problems are buried in the
OpenVMS bug lists, and probably a few postings around the 'net.
If you're using the DLM for your own resource coordination outside of C
(and not of something like PostgreSQL and its locking and I/O activities
via the C library), then you get to roll your own byte-range locking if
that's what you need. AFAIK, what OpenVMS C is doing there for
byte-range locking is specific to C applications, too. File and record
locks are general.
No, nothing for me. Just looking to perhaps help out the people who need such a
capability. One concept I'm solid on is that it should be a generic VMS
solution, not a C solution. But, that's how I am ... :-)
Post by Stephen Hoffman
As for application design, I've had success with just mapping the entire
file. That's not a typical OpenVMS design, but it does work nicely, and
it makes some tasks simpler. Reliability and crash recovery from
journals and/or from a shadowed server. Or using code to unmarshall the
data into memory, and then operating there, and marshalling the data
back out for a clean shutdown or such. Again, with a journal written to
storage or to another host or such, for purposes of crash recovery.
Maybe you'd be interested in contributing to the I/O side of things ???

:-)
Stephen Hoffman
2017-01-06 21:42:48 UTC
Permalink
Post by David Froble
Post by Stephen Hoffman
Not sure how you're envisioning I/O would be connected to the DLM.
I'm not. The DLM should be sort of an object that does a job.
However, there is nothing that I'm aware of that does byte range
locking, or has that need, that's currently implemented. Perhaps I
should ignore the I/O part of it and just concentrate on doing locks on
byte ranges.
Byte-range locking is "just" a convention of resource names added
within and used (solely) within the C RTL. Some details of the
resource names used were posted (somewhere) a while back IIRC, but
there's nothing published about the internal implementation within the
C RTL, AFAIK. RMS and the XQP are where the file system and
volume-level locking is implemented respectively, and the C RTL then
added some of its own lock-based coordination specifically for the
byte-range processing, as well as the buffer shuffling entailed for
that access as OpenVMS doesn't have a byte-range I/O API. The caller
— the C RTL in this case — has to shuffle the data around within its
own context. RMS I/O doesn't know from C byte-range locking, it (RMS)
has its own lock resource names and its own conventions. C uses its
own locks as well as having the usual and typical dependencies of other
RTL and application software on OpenVMS around the coordination
provided by lower-level software including RMS and the XQP and the lock
manager calls in those layers.

Somewhat related, there are some wrinkles here around what I/O
underneath C is atomic and what is not too, and that's what tripped up
the SSIO work.
Post by David Froble
Post by Stephen Hoffman
then the OpenVMS problems with shared stream I/O (SSIO) and a few other
details will arise. You've been involved in that discussion before,
though. For those that haven't been, there's an overview of SSIO (for
PostgreSQL or otherwise) at
http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf and
the discussions of SSIO-related bugs and problems are buried in the
OpenVMS bug lists, and probably a few postings around the 'net.
If you're using the DLM for your own resource coordination outside of C
(and not of something like PostgreSQL and its locking and I/O
activities via the C library), then you get to roll your own byte-range
locking if that's what you need. AFAIK, what OpenVMS C is doing there
for byte-range locking is specific to C applications, too. File and
record locks are general.
No, nothing for me. Just looking to perhaps help out the people who
need such a capability. One concept I'm solid on is that it should be
a generic VMS solution, not a C solution. But, that's how I am ...
:-)
OpenVMS I/O knows nothing of reading and writing ranges of bytes. No
support for that. OpenVMS I/O reads and writes records, and blocks.
This is part of why moving to byte-addressable storage — that's
probably two generations past what VSI is working on with VAFS or
whatever they're calling that new file system they're working on.
(First generation past is arguably a file system and I/O optimized for
block-addressable non-volatile storage, then for byte-addressable
non-volatile storage.)
Post by David Froble
Post by Stephen Hoffman
As for application design, I've had success with just mapping the
entire file. That's not a typical OpenVMS design, but it does work
nicely, and it makes some tasks simpler. Reliability and crash
recovery from journals and/or from a shadowed server. Or using code to
unmarshall the data into memory, and then operating there, and
marshalling the data back out for a clean shutdown or such. Again,
with a journal written to storage or to another host or such, for
purposes of crash recovery.
Maybe you'd be interested in contributing to the I/O side of things ???
:-)
I've already posted open source that does this.

As for digging the C RTL out from its present quagmire, I'm already on
record recommending rethinking and redesigning the whole tamalé, and
relegating the existing C RTL to retirement and eventual removal; of
replacing the current C RTL with a wholly new implementation. One
that does what's needed now and particularly going forward, that's
64-bit throughout, and preferably one that eliminates the workarounds,
hackery and endless compatibility shims and fixes and that replaces the
C RTL logical names as the control interface with something that works
rather more modularly, assuming that mechanism is even still needed.
--
Pure Personal Opinion | HoffmanLabs LLC
Craig A. Berry
2017-01-06 22:20:47 UTC
Permalink
Post by David Froble
Post by Stephen Hoffman
Not sure how you're envisioning I/O would be connected to the DLM.
I'm not. The DLM should be sort of an object that does a job.
However, there is nothing that I'm aware of that does byte range
locking, or has that need, that's currently implemented. Perhaps I
should ignore the I/O part of it and just concentrate on doing locks
on byte ranges.
Byte-range locking is "just" a convention of resource names added within
and used (solely) within the C RTL. Some details of the resource names
used were posted (somewhere) a while back IIRC, but there's nothing
published about the internal implementation within the C RTL, AFAIK.
RMS and the XQP are where the file system and volume-level locking is
implemented respectively, and the C RTL then added some of its own
lock-based coordination specifically for the byte-range processing, as
well as the buffer shuffling entailed for that access as OpenVMS doesn't
have a byte-range I/O API. The caller — the C RTL in this case — has
to shuffle the data around within its own context. RMS I/O doesn't
know from C byte-range locking, it (RMS) has its own lock resource names
and its own conventions. C uses its own locks as well as having the
usual and typical dependencies of other RTL and application software on
OpenVMS around the coordination provided by lower-level software
including RMS and the XQP and the lock manager calls in those layers.
Somewhat related, there are some wrinkles here around what I/O
underneath C is atomic and what is not too, and that's what tripped up
the SSIO work.
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC that bypasses all the stuff
described in your preceding paragraph.[1] It mystifies me why he would
want to start over from scratch rather than just encouraging VSI to fix
whatever kinks remain in SSIO and make it ready for prime time. He might
even advocate for the new API to be documented so it's available from
languages other than C :-).


[1] <http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf>
Stephen Hoffman
2017-01-06 23:58:17 UTC
Permalink
Post by Craig A. Berry
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC...
Works for me. (But then I'd like also to see a byte-addressable file
system, too.)

The existing byte-range locking and SSIO and C jackets is something
most of us would rather not have to deal with.

Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually using
interlocked queues for that sort of thing now, and that isn't always
the nicest approach for some of the designs.
--
Pure Personal Opinion | HoffmanLabs LLC
David Froble
2017-01-07 02:03:59 UTC
Permalink
Post by Stephen Hoffman
Post by Craig A. Berry
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC...
Works for me. (But then I'd like also to see a byte-addressable file
system, too.)
The existing byte-range locking and SSIO and C jackets is something most
of us would rather not have to deal with.
Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually using
interlocked queues for that sort of thing now, and that isn't always the
nicest approach for some of the designs.
An immediate question comes to mind. What is the magnitude of what you're
talking about? One question I've asked myself is, how big of integers might be
needed to define a range? Maybe quadword?

Yes, what I'm considering would be perfect for what you mention, and cluster
aware, if that's a consideration.

Any idea what language(s) were used to implement the DLM?
Stephen Hoffman
2017-01-07 04:15:24 UTC
Permalink
Post by David Froble
Post by Stephen Hoffman
Post by Craig A. Berry
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC...
Works for me. (But then I'd like also to see a byte-addressable file
system, too.)
The existing byte-range locking and SSIO and C jackets is something
most of us would rather not have to deal with.
Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually using
interlocked queues for that sort of thing now, and that isn't always
the nicest approach for some of the designs.
An immediate question comes to mind. What is the magnitude of what
you're talking about?
"It depends". Anything from a month or so for a DLM-interlocked
memory design and implementation and testing assuming a small and
well-specified and single-host requirement, on up to scheduling
infinity. Where in this range depending on what is and is not in
scope in the project, performance requirements and related, and how
compatible the design is to any existing source code and available
tools. It's a how-high-is-up right now.
Post by David Froble
One question I've asked myself is, how big of integers might be needed
to define a range? Maybe quadword?
In OpenVMS it'd be a quadword, as that's the size of the addresses
given current virtual addressing limit of OpenVMS. There's a a LCK
data structure pointing to that range block. Based on the definitions,
the relevant data structure appears to be named LKIDEF_BR, though that
structure does not appear to have been exposed in the Bliss or C
definitions for V8.4.
Post by David Froble
Yes, what I'm considering would be perfect for what you mention, and
cluster aware, if that's a consideration.
There'll be overhead shuffling the data around; it's basically
server-level mirroring. Performance is going to come down to the
speed of the host-to-host, and that's going to be a whole lot of
bespoke code for networking traffic quite possibly involving some of
the message queue software and DECdtm. RDMA would be handy here too,
certainly.

If the application requirements around performance permit it, I'd
deploy modern storage and 10 GbE networking and only cache the data in
memory when required. Might be able to brute-force the design and
avoid implementing mirrored memory, in other words. Flash and SSD
are massively faster than HDD, after all. That'd avoid a whole lot of
bespoke code, too. There's also the obvious and necessary discussion
around using one of the available commercial or open-source databases;
whether Oracle or otherwise.
--
Pure Personal Opinion | HoffmanLabs LLC
Bill Gunshannon
2017-01-07 16:49:48 UTC
Permalink
Post by David Froble
Post by Stephen Hoffman
Post by Craig A. Berry
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC...
Works for me. (But then I'd like also to see a byte-addressable
file system, too.)
The existing byte-range locking and SSIO and C jackets is something
most of us would rather not have to deal with.
Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually
using interlocked queues for that sort of thing now, and that isn't
always the nicest approach for some of the designs.
An immediate question comes to mind. What is the magnitude of what
you're talking about?
"It depends". Anything from a month or so for a DLM-interlocked memory
design and implementation and testing assuming a small and
well-specified and single-host requirement, on up to scheduling
infinity. Where in this range depending on what is and is not in scope
in the project, performance requirements and related, and how compatible
the design is to any existing source code and available tools. It's a
how-high-is-up right now.
Post by David Froble
One question I've asked myself is, how big of integers might be needed
to define a range? Maybe quadword?
In OpenVMS it'd be a quadword, as that's the size of the addresses given
current virtual addressing limit of OpenVMS. There's a a LCK data
structure pointing to that range block. Based on the definitions, the
relevant data structure appears to be named LKIDEF_BR, though that
structure does not appear to have been exposed in the Bliss or C
definitions for V8.4.
Post by David Froble
Yes, what I'm considering would be perfect for what you mention, and
cluster aware, if that's a consideration.
There'll be overhead shuffling the data around; it's basically
server-level mirroring. Performance is going to come down to the
speed of the host-to-host, and that's going to be a whole lot of bespoke
code for networking traffic quite possibly involving some of the message
queue software and DECdtm. RDMA would be handy here too, certainly.
If the application requirements around performance permit it, I'd deploy
modern storage and 10 GbE networking and only cache the data in memory
when required. Might be able to brute-force the design and avoid
implementing mirrored memory, in other words. Flash and SSD are
massively faster than HDD, after all. That'd avoid a whole lot of
bespoke code, too. There's also the obvious and necessary discussion
around using one of the available commercial or open-source databases;
whether Oracle or otherwise.
Hoff,
You skipped the one question I would have also considered
interesting.

"Any idea what language(s) were used to implement the DLM? "

bill
John Reagan
2017-01-07 17:27:42 UTC
Permalink
Post by Bill Gunshannon
Post by David Froble
Post by Stephen Hoffman
Post by Craig A. Berry
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC...
Works for me. (But then I'd like also to see a byte-addressable
file system, too.)
The existing byte-range locking and SSIO and C jackets is something
most of us would rather not have to deal with.
Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually
using interlocked queues for that sort of thing now, and that isn't
always the nicest approach for some of the designs.
An immediate question comes to mind. What is the magnitude of what
you're talking about?
"It depends". Anything from a month or so for a DLM-interlocked memory
design and implementation and testing assuming a small and
well-specified and single-host requirement, on up to scheduling
infinity. Where in this range depending on what is and is not in scope
in the project, performance requirements and related, and how compatible
the design is to any existing source code and available tools. It's a
how-high-is-up right now.
Post by David Froble
One question I've asked myself is, how big of integers might be needed
to define a range? Maybe quadword?
In OpenVMS it'd be a quadword, as that's the size of the addresses given
current virtual addressing limit of OpenVMS. There's a a LCK data
structure pointing to that range block. Based on the definitions, the
relevant data structure appears to be named LKIDEF_BR, though that
structure does not appear to have been exposed in the Bliss or C
definitions for V8.4.
Post by David Froble
Yes, what I'm considering would be perfect for what you mention, and
cluster aware, if that's a consideration.
There'll be overhead shuffling the data around; it's basically
server-level mirroring. Performance is going to come down to the
speed of the host-to-host, and that's going to be a whole lot of bespoke
code for networking traffic quite possibly involving some of the message
queue software and DECdtm. RDMA would be handy here too, certainly.
If the application requirements around performance permit it, I'd deploy
modern storage and 10 GbE networking and only cache the data in memory
when required. Might be able to brute-force the design and avoid
implementing mirrored memory, in other words. Flash and SSD are
massively faster than HDD, after all. That'd avoid a whole lot of
bespoke code, too. There's also the obvious and necessary discussion
around using one of the available commercial or open-source databases;
whether Oracle or otherwise.
Hoff,
You skipped the one question I would have also considered
interesting.
"Any idea what language(s) were used to implement the DLM? "
bill
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
Bill Gunshannon
2017-01-07 23:22:31 UTC
Permalink
Post by John Reagan
Post by Bill Gunshannon
Post by David Froble
Post by Stephen Hoffman
Post by Craig A. Berry
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC...
Works for me. (But then I'd like also to see a byte-addressable
file system, too.)
The existing byte-range locking and SSIO and C jackets is something
most of us would rather not have to deal with.
Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually
using interlocked queues for that sort of thing now, and that isn't
always the nicest approach for some of the designs.
An immediate question comes to mind. What is the magnitude of what
you're talking about?
"It depends". Anything from a month or so for a DLM-interlocked memory
design and implementation and testing assuming a small and
well-specified and single-host requirement, on up to scheduling
infinity. Where in this range depending on what is and is not in scope
in the project, performance requirements and related, and how compatible
the design is to any existing source code and available tools. It's a
how-high-is-up right now.
Post by David Froble
One question I've asked myself is, how big of integers might be needed
to define a range? Maybe quadword?
In OpenVMS it'd be a quadword, as that's the size of the addresses given
current virtual addressing limit of OpenVMS. There's a a LCK data
structure pointing to that range block. Based on the definitions, the
relevant data structure appears to be named LKIDEF_BR, though that
structure does not appear to have been exposed in the Bliss or C
definitions for V8.4.
Post by David Froble
Yes, what I'm considering would be perfect for what you mention, and
cluster aware, if that's a consideration.
There'll be overhead shuffling the data around; it's basically
server-level mirroring. Performance is going to come down to the
speed of the host-to-host, and that's going to be a whole lot of bespoke
code for networking traffic quite possibly involving some of the message
queue software and DECdtm. RDMA would be handy here too, certainly.
If the application requirements around performance permit it, I'd deploy
modern storage and 10 GbE networking and only cache the data in memory
when required. Might be able to brute-force the design and avoid
implementing mirrored memory, in other words. Flash and SSD are
massively faster than HDD, after all. That'd avoid a whole lot of
bespoke code, too. There's also the obvious and necessary discussion
around using one of the available commercial or open-source databases;
whether Oracle or otherwise.
Hoff,
You skipped the one question I would have also considered
interesting.
"Any idea what language(s) were used to implement the DLM? "
bill
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
Thank you.

bill
Stephen Hoffman
2017-01-08 14:50:31 UTC
Permalink
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.

Long answer:

That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.

The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.

As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.

I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.

What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.

As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.

Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...

There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.

The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.

VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.

I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)

In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.

https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Kerry Main
2017-01-08 16:02:36 UTC
Permalink
-----Original Message-----
Stephen Hoffman via Info-vax
Sent: January 8, 2017 9:51 AM
Subject: Re: [Info-vax] Internals and Data Structures - DLM information
On Saturday, January 7, 2017 at 11:49:51 AM UTC-5, Bill Gunshannon
Post by Bill Gunshannon
Hoff,
You skipped the one question I would have also considered
interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM.
Seemed
a good way to distract from what David was seeking, if I had
researched and had addressed it.
[snip..]
In summary.... Debating whether two-revisions-of-the-standard-back
C, or Macro32, or Bliss — or any other languages of similar vintage —
was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
Agree - imho, this reimplementation of tools discussion is analogous to someone looking at the Taj Mahal and thinking yes, it’s a great structure, but it would have been so much "better" if they had built it with "modern" tools.

Remember - Todays "modern" tools are tomorrows "fossils".

At some point in the future (X years), there will be references to Rust as a "fossil" in the same way as many today might refer to Bliss as a fossil. The only question is what is that value of "X".

What is important as a differentiator is not the tools something was built with, but rather the beauty (yes, there are always flaws) in what and how something was built.


Regards,

Kerry Main
Kerry dot main at starkgaming dot com
Stephen Hoffman
2017-01-08 20:00:13 UTC
Permalink
Post by Kerry Main
Todays "modern" tools are tomorrows "fossils".
People using tools from yesterday with depreciating values today are
excellent competitive targets tomorrow, too.

Good tools and good designs do hold their value longer than lesser
choices, but even good tools and good designs are often replaced by
better ones.

Be aware of what you're doing, and be aware of what others are doing.
You can't assume your tools and designs are still good, if you haven't
looked around.

As for the DLM, if it works for what you need, use it. Or look around
at the alternative approaches, and use one of those. There are many
papers and discussions...

http://www.mcs.anl.gov/~thakur/papers/byte-range.pdf
http://www.mcs.anl.gov/~thakur/papers/rma-fv.pdf
https://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/locking.html
http://fsfilters.blogspot.com/2011/11/byte-range-locks-and-irpmjcleanup.html
http://wiki.linux-nfs.org/wiki/index.php/Cluster_Coherent_NFS_and_Byte_Range_Locking

https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html
(issues with the POSIX locking APIs, etc. Includes "The Tar Pit:
Backwards Compatibility", too. Anybody need to guess where those share
modes came from?)
Etc...
--
Pure Personal Opinion | HoffmanLabs LLC
Kerry Main
2017-01-08 22:03:39 UTC
Permalink
-----Original Message-----
Stephen Hoffman via Info-vax
Sent: January 8, 2017 3:00 PM
Subject: Re: [Info-vax] Internals and Data Structures - DLM
information
Post by Kerry Main
Todays "modern" tools are tomorrows "fossils".
People using tools from yesterday with depreciating values today are
excellent competitive targets tomorrow, too.
It is also true that those breaking trail usually take the first
arrows ...
Good tools and good designs do hold their value longer than lesser
choices, but even good tools and good designs are often replaced by
better ones.
Replaced, yes .. but not necessarily "better".

One only has to look at how commodity OS's have, in many areas,
overcome tradition enterprise platforms (Solaris, AIX, OpenVMS,
NonStop etc.). Very few Customers who have migrated to commodity OS
platforms viewed them as "better" than their current platform, but
rather were cheaper (albeit debateable when looking at the big
picture, but that is the power of marketing).

I am sure every med to large company also have examples of business
applications which were replaced by corporate IT in the name of
"standardization", but in fact, provided only a fraction of the
flexibility and benefits of the "older" application.
Be aware of what you're doing, and be aware of what others are
doing.
You can't assume your tools and designs are still good, if you haven't
looked around.
Agree. Keeping current with trends is always good as long as one
understands that new does not necessarily mean better.
As for the DLM, if it works for what you need, use it. Or look around
at the alternative approaches, and use one of those. There are many
papers and discussions...
http://www.mcs.anl.gov/~thakur/papers/byte-range.pdf
http://www.mcs.anl.gov/~thakur/papers/rma-fv.pdf
https://www.samba.org/samba/docs/man/Samba-HOWTO-
Collection/locking.html
http://fsfilters.blogspot.com/2011/11/byte-range-locks-and-
irpmjcleanup.html
http://wiki.linux-
nfs.org/wiki/index.php/Cluster_Coherent_NFS_and_Byte_Range_Loc
king
https://www.samba.org/samba/news/articles/low_point/tale_two_st
ds_os2.html
Backwards Compatibility", too. Anybody need to guess where those
share modes came from?) Etc...
Interesting quote from the last url - another interesting reference to
where Windows "acquired" past technology from.
"However, the Win32 design does some things very well; security, for
instance. Security isn't the number one thing people think of when
considering Windows, but in the Win32 API security is a very great
concern. In Win32, every object can be secured, and a property called
a "Security Descriptor" which contains an access control list (ACL)
can be attached to it. This means objects like processes, files,
directories, even Windows can have ACLs attached. This is much cleaner
than POSIX, where only objects in the file system can have ACLs
attached to them."

As for the DLM in general, one needs to remember there are different
clustering models in play. Windows/UNIX/Linux/NonStop are primarily
shared nothing models whereas OpenVMS, Linux/GFS (file system limit
100TB) and z/OS are shared disk models.

Both clustering models have pro's and con's, but here is a WP which
explains the pro's and con's of each model:

Shortened url:
http://bit.ly/2dScx9k
"Comparing shared-nothing and shared-disk in benchmarks is analogous
to comparing a dragster and a Porsche. The dragster, like the
hand-tuned shared-nothing database, will beat the Porsche in a
straight quarter mile race. However, the Porsche, like a shared-disk
database, will easily beat the dragster on regular roads. If your
selected benchmark is a quarter mile straightaway that tests all out
speed, like Sysbench, a shared-nothing database will win. However,
shared-disk will perform better in real world environments."

Original (will wrap)
http://www.scaledb.com/wp-content/uploads/2015/11/Shared-Nohing-vs-Sha
red-Disk-WP_SDvSN.pdf

So, do you need a dragster or a Porsche?

:-)

Regards,

Kerry Main
Kerry dot main at starkgaming dot com
Simon Clubley
2017-01-09 00:53:13 UTC
Permalink
Post by Kerry Main
At some point in the future (X years), there will be references to
Rust as a "fossil" in the same way as many today might refer to Bliss
as a fossil. The only question is what is that value of "X".
And when that happens, there will still be people using Ada in
safety critical applications. Ada was targetted for a specific
type of high reliability requirement and it does that very well.

(Unfortunately for general system security, there will also still
be people writing low level code in C when that happens because of
the massive installed base and because it still satisfies a certain
technology and (low level) portability requirement.)

BTW, I've tried looking at Rust, but it's syntax is _really_ yucky
to work with and I can't help thinking that looking at Rust code
6 months to a year later would not exactly be a great experience.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
Bill Gunshannon
2017-01-09 01:21:34 UTC
Permalink
Post by Simon Clubley
Post by Kerry Main
At some point in the future (X years), there will be references to
Rust as a "fossil" in the same way as many today might refer to Bliss
as a fossil. The only question is what is that value of "X".
And when that happens, there will still be people using Ada in
safety critical applications. Ada was targetted for a specific
type of high reliability requirement and it does that very well.
And it missed the mark by many a mile. Ask the Air Force. It was
their idea in the first place but by the time it became a reality
it was so different from what they wanted they refused to use it
even when DOD tried to mandate it. Last I checked they were still
using Jovial out in Dayton.
Post by Simon Clubley
(Unfortunately for general system security, there will also still
be people writing low level code in C when that happens because of
the massive installed base and because it still satisfies a certain
technology and (low level) portability requirement.)
Back in my government contracting days my employer was big on using
Ada. I was a member of the company's Ada User Group and once mentioned
to one of the major experts that I thought it would be interesting
to re-write the C parts of Unix in Ada. I was informed that the result
would be a totally unusable OS. The code would be too inefficient
to be practical.
Post by Simon Clubley
BTW, I've tried looking at Rust, but it's syntax is _really_ yucky
to work with and I can't help thinking that looking at Rust code
6 months to a year later would not exactly be a great experience.
Sounds like PHP. I think I need to take a look at Rust. I haven't
had a good laugh all week.

bill
Scott Dorsey
2017-01-09 02:21:39 UTC
Permalink
Post by Bill Gunshannon
Back in my government contracting days my employer was big on using
Ada. I was a member of the company's Ada User Group and once mentioned
to one of the major experts that I thought it would be interesting
to re-write the C parts of Unix in Ada. I was informed that the result
would be a totally unusable OS. The code would be too inefficient
to be practical.
Ada was the PL/1 of the nineties, with every possible feature added to the
language long before people had figured out how to make compilers work with
all those features.

So first you have the issue where every programmer is using his own personal
subset of the language because the language is too big for everyone to know
everything, so people can't necessarily read one another's code.

And secondly you have the issue of the compiler requiring tremendous resources
and optimization being difficult. In 1989 we had a Data General machine that
could support about 100 developers writing Pascal, or two developers writing
Ada.

Fortunately times have changed and the efficiency issues of the runtime code
has been fixed, while the compilation efficiency is less of a problem due to
much faster hardware.

But the issue with programmers not knowing the whole thing remains.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."
c***@gmail.com
2017-01-08 21:54:58 UTC
Permalink
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Trivia time. In the most recent Itanium system build the breakdown of 8756 modules by language is:

Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%

These totals do NOT include the loader, layered products, compilers, DECnet V, TCPIP, and DECwindows.
Bill Gunshannon
2017-01-08 22:42:05 UTC
Permalink
Post by c***@gmail.com
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers, DECnet V, TCPIP, and DECwindows.
Ada? Doing what? :-)

And, I thought MACRO was the VMS Assembler. Is there another?

bill
c***@gmail.com
2017-01-08 22:52:08 UTC
Permalink
Post by Bill Gunshannon
Post by c***@gmail.com
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers, DECnet V, TCPIP, and DECwindows.
Ada? Doing what? :-)
And, I thought MACRO was the VMS Assembler. Is there another?
bill
Two security-related facilities are written mostly in Ada.

The MACRO compiler compiles the VAX MACRO-32 code for Itanium.
Simon Clubley
2017-01-09 00:39:31 UTC
Permalink
Post by c***@gmail.com
Two security-related facilities are written mostly in Ada.
Well done to whoever did that. (I'm being serious here.)

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
Stephen Hoffman
2017-01-09 15:56:47 UTC
Permalink
Post by c***@gmail.com
Post by Bill Gunshannon
Post by c***@gmail.com
Trivia time. In the most recent Itanium system build the breakdown of
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers,
DECnet V, TCPIP, and DECwindows.
...
And, I thought MACRO was the VMS Assembler. Is there another?
...
...
The MACRO compiler compiles the VAX MACRO-32 code for Itanium.
I suspect Bill is wondering why "MACRO" — likely the VAX Macro32
compiler — and "Assembler" — likely ias in this context, the Itanium
native assembler — are called out separately in the list.
--
Pure Personal Opinion | HoffmanLabs LLC
Jan-Erik Soderholm
2017-01-08 23:49:38 UTC
Permalink
Post by Bill Gunshannon
Post by c***@gmail.com
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Trivia time. In the most recent Itanium system build the breakdown of
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers,
DECnet V, TCPIP, and DECwindows.
Ada? Doing what? :-)
And, I thought MACRO was the VMS Assembler. Is there another?
bill
Macro-32 was the "Macro Assembler" on VAX where you wrote VAX
machine code mnemonics.

On Alpha and Itanium Macro-32 are compilers, compiling the VAX
instructions into Alpha or Itanium machine code.

Then there are native assamblers, on Alpha it is called Macro-64.
It was a separate kit and not delivered with the base OS (such as
MACRO32 is). Seems to be available from the freeware V4 CD:
http://www.decuslib.com/freeware/freewarev40/macro64/

The Itanium assembler seems to be a port of the open source Intel
Itanium version called "IAS". I found this:
http://decuslib.com/HP_Open_Source_Tools/ias/kit/
There is a readme file describing the "kit".
David Froble
2017-01-09 05:18:02 UTC
Permalink
Post by Bill Gunshannon
Post by c***@gmail.com
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Trivia time. In the most recent Itanium system build the breakdown of
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers,
DECnet V, TCPIP, and DECwindows.
Ada? Doing what? :-)
And, I thought MACRO was the VMS Assembler. Is there another?
bill
The Assembler 1% is most likely itanic assembly code.

Looks like John is getting his way, the use of Macro-32 is shrinking. If the
compiler ever goes away I've got a database product that will be going extinct.
c***@gmail.com
2017-01-09 11:31:22 UTC
Permalink
Post by David Froble
Post by Bill Gunshannon
Post by c***@gmail.com
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Trivia time. In the most recent Itanium system build the breakdown of
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers,
DECnet V, TCPIP, and DECwindows.
Ada? Doing what? :-)
And, I thought MACRO was the VMS Assembler. Is there another?
bill
The Assembler 1% is most likely itanic assembly code.
Looks like John is getting his way, the use of Macro-32 is shrinking. If the
compiler ever goes away I've got a database product that will be going extinct.
Guess I should have been more precise. This was an Itanium build so

assembler = Itanium assembler

MACRO = VAX MACRO-32
Stephen Hoffman
2017-01-09 15:52:07 UTC
Permalink
Post by c***@gmail.com
Post by Stephen Hoffman
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
Trivia time. In the most recent Itanium system build the breakdown of
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers,
DECnet V, TCPIP, and DECwindows.
Thanks for posting that.

The KLOC tools measured usage based on the numbers of lines of code,
rather than the numbers of source modules.

Every source code control system eventually develops these metadata
tools; the KLOC stuff or git-extras or otherwise.

All sorts of meaningless metrics and statistics are possible here, too.
More than a few places collect and use this data.

Worse — whether lines of code or module counts or bug reports closed or
sales commissions — you tend to get what you measure.

One of my favorite tales of this morass:
http://www.folklore.org/StoryView.py?story=Negative_2000_Lines_Of_Code.txt

I'd personally prefer to see less of C, Bliss and Macro32 code solely
because of the problems of each of those three.

And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
efficient system-level coding.

See how we're drifting away from the DLM and byte-range DLM distributed
limerick managment discussion, too?
--
Pure Personal Opinion | HoffmanLabs LLC
Simon Clubley
2017-01-09 18:33:22 UTC
Permalink
Post by Stephen Hoffman
And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
efficient system-level coding.
You are right about the first three, but you are way wrong about Ada.

Ada has various language features for low level access and Ada
compilers allow you to compile for various levels of runtime support
so you can even run Ada at bare metal level without RTL support if
you wish.

The ARG has even been working on a couple of AIs I submitted a couple
of years ago to enhance Ada's abilities in this area so you can more
easily use Ada types with device registers instead of opaque integers
and C style bitmasks.

One of those was ratified by WG9 in October (ironically, the one
which I didn't think stood a chance of being passed :-)) and the
main one is an active ARG work item.

My main complaint with the Ada ecosystem is the lack of freely
available Ada compilers for all the systems that have freely
available C compilers. It is not about the language itself,
although I have sometimes wondered if some kind of intermediate
systems level programming language (such as what DEC tried with
Pillar) might make it easier to ease people into the discipline
required when writing Ada code.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
j***@yahoo.co.uk
2017-01-09 21:33:15 UTC
Permalink
Post by Simon Clubley
Post by Stephen Hoffman
And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
efficient system-level coding.
You are right about the first three, but you are way wrong about Ada.
Ada has various language features for low level access and Ada
compilers allow you to compile for various levels of runtime support
so you can even run Ada at bare metal level without RTL support if
you wish.
The ARG has even been working on a couple of AIs I submitted a couple
of years ago to enhance Ada's abilities in this area so you can more
easily use Ada types with device registers instead of opaque integers
and C style bitmasks.
One of those was ratified by WG9 in October (ironically, the one
which I didn't think stood a chance of being passed :-)) and the
main one is an active ARG work item.
My main complaint with the Ada ecosystem is the lack of freely
available Ada compilers for all the systems that have freely
available C compilers. It is not about the language itself,
although I have sometimes wondered if some kind of intermediate
systems level programming language (such as what DEC tried with
Pillar) might make it easier to ease people into the discipline
required when writing Ada code.
Simon.
--
Microsoft: Bringing you 1980s technology to a 21st century world
Agreed.

Ada is a decent language to design in. It can even require
people to think before stuff will even compile properly.
Obviously that requirement is very unhelpful in 'IT' in
recent years, which is entirely unrelated to why so many
IT projects *still* fail to meet budgets and/or
timescales, and/or fail in service.

Given a sufficiently skilled developer team and a
sufficiently adequate compilation system (there's more to
Ada than a compiler), it's even a decent language to
develop some kinds of programs in, including the kind of
low level stuff more normally done in C. These dodgier
things can be done, it will look awkward, and to those
with a clue, it shouts out a million miles that "this
code may be high risk".

The language design, and the then-typical state of the art
of compiler development, was such that many Ada compilers
were pretty rubbish at producing error messages (hmmm,
which other 'popular' language's 'popular' compiler(s)
often provokes similar-sounding complaints...).

A couple of Ada users I knew kept their code compatible
with DEC Ada even though the deployment platform was SPARC,
and kept a VMS system on hand with DEC Ada, just because
of the quality of the DEC Ada compiler error messages.

Then someone in HQ decided that there was no money in
updating DEC Ada to something more recent, and that
GreenHills or GNAT would be more than adequate. Oh well.
David Froble
2017-01-09 21:13:18 UTC
Permalink
Post by Stephen Hoffman
Post by c***@gmail.com
Post by Stephen Hoffman
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the
statistical reporting tools in the OpenVMS source code control system
— not since I last did that ~twenty years ago.
Trivia time. In the most recent Itanium system build the breakdown of
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers,
DECnet V, TCPIP, and DECwindows.
Thanks for posting that.
The KLOC tools measured usage based on the numbers of lines of code,
rather than the numbers of source modules.
Every source code control system eventually develops these metadata
tools; the KLOC stuff or git-extras or otherwise.
All sorts of meaningless metrics and statistics are possible here, too.
More than a few places collect and use this data.
Worse — whether lines of code or module counts or bug reports closed or
sales commissions — you tend to get what you measure.
http://www.folklore.org/StoryView.py?story=Negative_2000_Lines_Of_Code.txt
I'd personally prefer to see less of C, Bliss and Macro32 code solely
because of the problems of each of those three.
Honest question here, as I've never used Bliss. Wasn't it developed with system
level code as the main target? If so, why is it un-preferred? Is this a case
of greener grass elsewhere from where one's current location?
Post by Stephen Hoffman
And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
efficient system-level coding.
And here we have the issue of compiler implementation vs applicability. Basic
for one is not appropriate for system level code because most of it's activity
uses RTL routines, which aren't appropriate for system level code. Or at least
some of it. However, if a Basic compiler were implemented that was targetted at
kernel mode code, threads, and such, perhaps the actual language might be a
decent choice.

Nah, too easy to read ...

Actually, I've seen Basic code that was rather un-readable, for me at least.

Now, if the statement was "Basic, Fortran, and Cobol as implemented on VMS are
not viable solutions for system-level coding", I'll buy that.
Post by Stephen Hoffman
See how we're drifting away from the DLM and byte-range DLM distributed
limerick managment discussion, too?
Ditch the "limerick" joke. I think we've been stuck with "lock" for way too
long to change now. Besides, an accurate description would be way too wordy.
Michael Moroney
2017-01-09 23:06:20 UTC
Permalink
Post by David Froble
Ditch the "limerick" joke. I think we've been stuck with "lock" for way too
long to change now. Besides, an accurate description would be way too wordy.
I think a nice short description might be to compare it to a traffic light.
A traffic light tells you when to go or stop, but does nothing to enforce
it (unlike, say, a railroad crossing which will at least try to stop you
with a crossing gate). It's up to the desire not to get into a wreck, and
cops writing tickets, that causes drivers to obey the light.

Yeah, yeah, some traffic lights will take your picture and send you a
ticket, ignore those and just consider the old fashioned ones on a timer.
David Froble
2017-01-10 01:05:11 UTC
Permalink
Post by Michael Moroney
Post by David Froble
Ditch the "limerick" joke. I think we've been stuck with "lock" for way too
long to change now. Besides, an accurate description would be way too wordy.
I think a nice short description might be to compare it to a traffic light.
A traffic light tells you when to go or stop, but does nothing to enforce
it (unlike, say, a railroad crossing which will at least try to stop you
with a crossing gate). It's up to the desire not to get into a wreck, and
cops writing tickets, that causes drivers to obey the light.
Yeah, yeah, some traffic lights will take your picture and send you a
ticket, ignore those and just consider the old fashioned ones on a timer.
And just look at how many words you needed to use. :-)

Best to just stick with "lock", most of us understand what it really does.

:-)

Better too than getting into discussions of running red lights and busting
through crossing gates ....
Stephen Hoffman
2017-01-10 00:16:20 UTC
Permalink
Post by David Froble
Honest question here, as I've never used Bliss. Wasn't it developed
with system level code as the main target? If so, why is it
un-preferred? Is this a case of greener grass elsewhere from where
one's current location?
The OpenVMS development decision was based on what most of the industry
was using for system code, and that was C and C++. Not Bliss.

Looking forward, and why I'm less than fond of it: Bliss has problems
akin to those of C around pointers and type punning and memory
management and the ease of crashing around unexpected null pointers and
such. This is part of why I'm less than fond of all three; of system
code written in Bliss, C or Macro32.

Looking backward: DEC priced Bliss completely out of the market decades
ago, even among OpenVMS sites. Which had the inevitable result; of
the pool of Bliss programmers and Bliss compilers and tools and open
source and third-party projects using Bliss being... a small one.
Which means you get to create and maintain all your own tooling, and
staff training is more involved. Which adds costs.

Looking at project costs: Adding languages adds incremental costs to
your development efforts, so you either need to reap substantial
benefits from any added language, or you need to slowly migrate your
old code and tooling over — and which isn't a very effective use of
time, and tends to work best when you can deprecate the old hunks — or
you make due and try to make the tools you're using work as well as you
can and hope you don't miss out if (when?) somebody gains an edge from
the efficiency of the tools they're using.

Considerations by and of the staff: the flip side of the relatively
small pool of Bliss developers means that the developers you have and
that you have trained aren't themselves going to be as interesting to
the rest of the industry, and that tends to limit the job choices that
these Bliss developer have available now and in the future. Bliss
just isn't a skill that's particularly transportable nor overly popular
among developers nor among prospective new employers. The pool of
folks that want to learn Bliss isn't large, either.
Post by David Froble
Post by Stephen Hoffman
And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
efficient system-level coding.
And here we have the issue of compiler implementation vs applicability.
Basic for one is not appropriate for system level code because most of
it's activity uses RTL routines, which aren't appropriate for system
level code. Or at least some of it. However, if a Basic compiler were
implemented that was targetted at kernel mode code, threads, and such,
perhaps the actual language might be a decent choice.
Sure, but then you've invented your own Bliss. Could you do this?
Sure. But you're left to deal with language syntax design messes, and
differences in how traditional BASIC handles memory allocations and how
system code tends to want to have that work. You'll likely want
multithreading support in the language, too. Among other details.

As for Rust, there are various write-ups that might help understand
some of the motivations.
http://www.oreilly.com/programming/free/files/why-rust.pdf Rust is
also really easy to call into C code, too.

Go is one of the newer languages that's intended for tasks similar to
what you'd use BASIC for. Probably not the best choice for writing
system-level code, though.

But then I don't see VSI adding a fourth language for OpenVMS system
programming to the existing Macro32, C and Bliss usage, and migrating
substantial quantities of the existing code away from one of those
three languages — or translating from Ada or any other existing OpenVMS
code over to Rust for that matter — is vanishingly unlikely.
--
Pure Personal Opinion | HoffmanLabs LLC
John Reagan
2017-01-10 15:20:44 UTC
Permalink
Post by Stephen Hoffman
Bliss
just isn't a skill that's particularly transportable nor overly popular
among developers nor among prospective new employers. The pool of
folks that want to learn Bliss isn't large, either.
As VSI's BLISS "artist in residence" I'm devastated. :)

Seriously, at this year's LLVM fall developers conference, somebody saw the "VMS Software" logo on my fleece (Sue gets us good swag) and come up to me to talk about OpenVMS. As I discussed the plans for using LLVM, his eyes got big and he said "You mean you'll have a BLISS compiler for x86? Cool!"
Scott Dorsey
2017-01-10 15:34:54 UTC
Permalink
Post by John Reagan
Bliss=20
just isn't a skill that's particularly transportable nor overly popular=
=20
among developers nor among prospective new employers. The pool of=20
folks that want to learn Bliss isn't large, either.
As VSI's BLISS "artist in residence" I'm devastated. :)
I still have the "BLISS IS IGNORANCE" T-shirt that was being handed out
at DECUS...
Post by John Reagan
Seriously, at this year's LLVM fall developers conference, somebody saw the=
"VMS Software" logo on my fleece (Sue gets us good swag) and come up to me=
to talk about OpenVMS. As I discussed the plans for using LLVM, his eyes =
got big and he said "You mean you'll have a BLISS compiler for x86? Cool!"
Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."
Bill Gunshannon
2017-01-10 15:54:17 UTC
Permalink
Post by Scott Dorsey
Post by John Reagan
Bliss=20
just isn't a skill that's particularly transportable nor overly popular=
=20
among developers nor among prospective new employers. The pool of=20
folks that want to learn Bliss isn't large, either.
As VSI's BLISS "artist in residence" I'm devastated. :)
I still have the "BLISS IS IGNORANCE" T-shirt that was being handed out
at DECUS...
Post by John Reagan
Seriously, at this year's LLVM fall developers conference, somebody saw the=
"VMS Software" logo on my fleece (Sue gets us good swag) and come up to me=
to talk about OpenVMS. As I discussed the plans for using LLVM, his eyes =
got big and he said "You mean you'll have a BLISS compiler for x86? Cool!"
Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
--scott
How about someone writing a BLISS to C converter? That's how the Open
Source Compilers for Ada, Fortran, Pascal and COBOL all started. :-)

bill
Scott Dorsey
2017-01-10 16:01:23 UTC
Permalink
Post by Bill Gunshannon
How about someone writing a BLISS to C converter? That's how the Open
Source Compilers for Ada, Fortran, Pascal and COBOL all started. :-)
I can't speak for the others, but in the case of g77, the f2c front end
nonsense delayed getting a decent working fortran compiler by nearly two
decades.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."
John Reagan
2017-01-10 16:43:23 UTC
Permalink
Post by Bill Gunshannon
Post by Scott Dorsey
Post by John Reagan
Bliss=20
just isn't a skill that's particularly transportable nor overly popular=
=20
among developers nor among prospective new employers. The pool of=20
folks that want to learn Bliss isn't large, either.
As VSI's BLISS "artist in residence" I'm devastated. :)
I still have the "BLISS IS IGNORANCE" T-shirt that was being handed out
at DECUS...
Post by John Reagan
Seriously, at this year's LLVM fall developers conference, somebody saw the=
"VMS Software" logo on my fleece (Sue gets us good swag) and come up to me=
to talk about OpenVMS. As I discussed the plans for using LLVM, his eyes =
got big and he said "You mean you'll have a BLISS compiler for x86? Cool!"
Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
--scott
How about someone writing a BLISS to C converter? That's how the Open
Source Compilers for Ada, Fortran, Pascal and COBOL all started. :-)
bill
You mean "readable", "maintainable", and "debuggable" C from BLISS? You'll be standing on top of all the corpses that have tried before you. Easy programs are easy, complex BLISS modules which rely on the BLISS macro facility approach "impossible". Even the Holy Hand Grenade of Antioch won't help.

And a tip-o-the-hat to Rust, it has a very powerful macro language that actually might come close to handling BLISS' feature set.
Bill Gunshannon
2017-01-10 16:55:03 UTC
Permalink
Post by John Reagan
Post by Bill Gunshannon
Post by Scott Dorsey
Post by John Reagan
Bliss=20
just isn't a skill that's particularly transportable nor overly popular=
=20
among developers nor among prospective new employers. The pool of=20
folks that want to learn Bliss isn't large, either.
As VSI's BLISS "artist in residence" I'm devastated. :)
I still have the "BLISS IS IGNORANCE" T-shirt that was being handed out
at DECUS...
Post by John Reagan
Seriously, at this year's LLVM fall developers conference, somebody saw the=
"VMS Software" logo on my fleece (Sue gets us good swag) and come up to me=
to talk about OpenVMS. As I discussed the plans for using LLVM, his eyes =
got big and he said "You mean you'll have a BLISS compiler for x86? Cool!"
Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
--scott
How about someone writing a BLISS to C converter? That's how the Open
Source Compilers for Ada, Fortran, Pascal and COBOL all started. :-)
bill
You mean "readable", "maintainable", and "debuggable" C from BLISS?
No, doesn't have to be "readable", "maintainable", or "debuggable". I
can't think of anyone who ever worked with the C output from any of
the products I mentioned who wasn't a compiler developer trying to
debug the translator. As it is with GnuCOBOL the C is merely an
intermediate language for the machine.
Post by John Reagan
You'll be standing on top of
all the corpses that have tried before you. Easy programs are easy, complex BLISS modules which
rely on the BLISS macro facility approach "impossible". Even the Holy Hand Grenade of Antioch won't help.
I can remember when people said converting COBOL to C was impossible,
too. And yet, there have been two completely functional examples that
it wasn't.
Post by John Reagan
And a tip-o-the-hat to Rust, it has a very powerful macro language that actually might come close
to handling BLISS' feature set.
I really need to look at this Rust thing. But I doubt I will
be as impressed as some of the people here seem to be.

bill
Scott Dorsey
2017-01-10 17:06:08 UTC
Permalink
Post by Bill Gunshannon
No, doesn't have to be "readable", "maintainable", or "debuggable". I
can't think of anyone who ever worked with the C output from any of
the products I mentioned who wasn't a compiler developer trying to
debug the translator. As it is with GnuCOBOL the C is merely an
intermediate language for the machine.
With g77, I did it all the time. Constantly. People would bring me in
code that didn't give the right answers and after the first step of making
sure it wasn't a user error, I wound up regularly spending time trying to
figure out why the translator was doing what it was.

It was horrible, horrible, horrible. So many different stages for things
to go awry. I tried to get users onto Intel or Portland Group compilers
or even the NagWare compiler as much as I could.

When gfortran came along, the first few releases were pretty full of bugs,
but it was comparatively easy to find and fix them, unlike g77. After the
first year or so of fooling with it, it was working better than g77 ever did.

The whole f2c nonsense was a temporary thing while the real fortran compiler
was developed, except that it turned out there wasn't any money or enough
people to develop the real fortran compiler, leaving us all with g77 until
2007 or so.
Post by Bill Gunshannon
I can remember when people said converting COBOL to C was impossible,
too. And yet, there have been two completely functional examples that
it wasn't.
I can say nothing about the COBOL translator, having not used it. The thing
about COBOL is that the difficult parts and the most important parts of typical
COBOL programs have nothing to do with the actual "procedure division" code
itself. How hard that is to deal with I don't know.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."
V***@SendSpamHere.ORG
2017-01-10 20:01:38 UTC
Permalink
Post by Bill Gunshannon
Post by Scott Dorsey
Post by John Reagan
Bliss=3D20
just isn't a skill that's particularly transportable nor overly popul=
ar=3D
Post by Bill Gunshannon
Post by Scott Dorsey
Post by John Reagan
=3D20
among developers nor among prospective new employers. The pool of=3D=
20
Post by Bill Gunshannon
Post by Scott Dorsey
Post by John Reagan
folks that want to learn Bliss isn't large, either.
As VSI's BLISS "artist in residence" I'm devastated. :)
I still have the "BLISS IS IGNORANCE" T-shirt that was being handed out
at DECUS...
Post by John Reagan
Seriously, at this year's LLVM fall developers conference, somebody sa=
w the=3D
Post by Bill Gunshannon
Post by Scott Dorsey
Post by John Reagan
"VMS Software" logo on my fleece (Sue gets us good swag) and come up t=
o me=3D
Post by Bill Gunshannon
Post by Scott Dorsey
Post by John Reagan
to talk about OpenVMS. As I discussed the plans for using LLVM, his e=
yes =3D
Post by Bill Gunshannon
Post by Scott Dorsey
Post by John Reagan
got big and he said "You mean you'll have a BLISS compiler for x86? Co=
ol!"
Post by Bill Gunshannon
Post by Scott Dorsey
Now, offer a free BLISS port for Linux and you might get some users. P=
robably
Post by Bill Gunshannon
Post by Scott Dorsey
not enough users to make up for the cost of making a free BLISS port to=
Linux,
Post by Bill Gunshannon
Post by Scott Dorsey
though.
--scott
=20
How about someone writing a BLISS to C converter? That's how the Open
Source Compilers for Ada, Fortran, Pascal and COBOL all started. :-)
=20
bill
You mean "readable", "maintainable", and "debuggable" C from BLISS? You'll=
be standing on top of all the corpses that have tried before you. Easy pr=
ograms are easy, complex BLISS modules which rely on the BLISS macro facili=
ty approach "impossible". Even the Holy Hand Grenade of Antioch won't help=
..
I'd rather read the Bliss in VMS than the C -- with or without Brother Maynard
by my side.
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
John Reagan
2017-01-10 16:45:00 UTC
Permalink
Post by Scott Dorsey
Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
Another reminder that VSI doesn't OWN the BLISS frontend code so I'm unable to create such a compiler even if I wanted to (and I want to) without permission from HPE.
Scott Dorsey
2017-01-10 16:57:52 UTC
Permalink
Post by John Reagan
Post by Scott Dorsey
Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
Another reminder that VSI doesn't OWN the BLISS frontend code so I'm unable to create such a compiler even if I wanted to (and I want to) without permission from HPE.
Is the frontend code even useful any longer, in this modern day of automatic
parser generators and LLVM and the like? I think if I wanted to build a
compiler to make x86 ELF files, I'd just take the gcc code generator and llvm
the rest of it.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."
John Reagan
2017-01-10 17:58:37 UTC
Permalink
In article <>,
Post by John Reagan
Post by Scott Dorsey
Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
Another reminder that VSI doesn't OWN the BLISS frontend code so I'm unable to create such a compiler even if I wanted to (and I want to) without permission from HPE.
Is the frontend code even useful any longer, in this modern day of automatic
parser generators and LLVM and the like? I think if I wanted to build a
compiler to make x86 ELF files, I'd just take the gcc code generator and llvm
the rest of it.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."
Why not LLVM's code generator? I see it as an either/or situation and LLVM's license is less restrictive.

As for whether the BLISS frontend contains any value and could be open sourced, that would have to be HPE's decision. Given that there is no support money today for BLISS compilers and we have BLISS compilers on the freeware CDs, it would be an easy choice for me. The BLISS frontend uses GEM shell routines for command line processing, error message generation, listing file management, etc. that would pull in some of GEM, etc. That would also need a Linux version (since GEM ran on Tru64), such things sorta exist down in our CMS libraries if you dig deep enough. We even had a Tru64 version of a portable MESSAGE compiler.
Simon Clubley
2017-01-10 18:58:07 UTC
Permalink
Post by Scott Dorsey
Is the frontend code even useful any longer, in this modern day of automatic
parser generators and LLVM and the like? I think if I wanted to build a
compiler to make x86 ELF files, I'd just take the gcc code generator and llvm
the rest of it.
Should that be the other way around, because LLVM essentially _is_
the code generator part of the compiler and you supply the frontend
code to LLVM.

The last attempt to use the gcc frontends BTW was called DragonEgg
and it didn't end well as the project fell into disuse and doesn't
work with current LLVM versions.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
David Froble
2017-01-10 19:38:01 UTC
Permalink
Post by John Reagan
Post by Scott Dorsey
Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
Another reminder that VSI doesn't OWN the BLISS frontend code so I'm unable to create such a compiler even if I wanted to (and I want to) without permission from HPE.
You just need to be a better negotiator. HP has come to you for some things in
the past, I believe. Should they do so again, add a few more demands onto your
already substantial renumeration.

:-)

I really cannot see what HP would lose by giving up the exclusive rights to
something they probably don't even want.
Johnny Billquist
2017-01-10 19:56:27 UTC
Permalink
Post by David Froble
Post by John Reagan
Post by Scott Dorsey
Now, offer a free BLISS port for Linux and you might get some users.
Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
Another reminder that VSI doesn't OWN the BLISS frontend code so I'm
unable to create such a compiler even if I wanted to (and I want to)
without permission from HPE.
You just need to be a better negotiator. HP has come to you for some
things in the past, I believe. Should they do so again, add a few more
demands onto your already substantial renumeration.
:-)
I really cannot see what HP would lose by giving up the exclusive rights
to something they probably don't even want.
Ask that question again, but substitute "PDP-11 Software" for "BLISS",
and see how far you get... :-/

Johnny
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: ***@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
Stephen Hoffman
2017-01-10 19:10:11 UTC
Permalink
Post by John Reagan
Bliss just isn't a skill that's particularly transportable nor overly
popular among developers nor among prospective new employers. The pool
of folks that want to learn Bliss isn't large, either.
As VSI's BLISS "artist in residence" I'm devastated. :)
Oh but John, you know you must stay true to your own inner Wirth. 😉
Post by John Reagan
Seriously, at this year's LLVM fall developers conference, somebody saw
the "VMS Software" logo on my fleece (Sue gets us good swag) and come
up to me to talk about OpenVMS. As I discussed the plans for using
LLVM, his eyes got big and he said "You mean you'll have a BLISS
compiler for x86? Cool!"
For those interested in following their own Bliss:

https://harlanhaskins.com/2017/01/08/building-a-compiler-with-swift-in-llvm-part-1-introduction-and-the-lexer.html
--
Pure Personal Opinion | HoffmanLabs LLC
Simon Clubley
2017-01-10 18:44:04 UTC
Permalink
Post by Stephen Hoffman
Go is one of the newer languages that's intended for tasks similar to
what you'd use BASIC for. Probably not the best choice for writing
system-level code, though.
Ah, yes, the language which enforces the K&R brace style. :-)

(I'm a Whitesmiths person myself BTW).

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
V***@SendSpamHere.ORG
2017-01-09 23:55:54 UTC
Permalink
Post by Bill Gunshannon
Post by c***@gmail.com
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers, DECnet V, TCPIP, and DECwindows.
Ada? Doing what? :-)
And, I thought MACRO was the VMS Assembler. Is there another?
bill
On Alpha and Itanium, Macro is compiled. Both Alpha and Itanium have native assemblers.
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
David Froble
2017-01-10 01:10:22 UTC
Permalink
Post by V***@SendSpamHere.ORG
Post by Bill Gunshannon
Post by c***@gmail.com
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers, DECnet V, TCPIP, and DECwindows.
Ada? Doing what? :-)
And, I thought MACRO was the VMS Assembler. Is there another?
bill
On Alpha and Itanium, Macro is compiled. Both Alpha and Itanium have native assemblers.
Ayep! And, if you got any smarts at all, you'll run, fast, screaming, from
either one.

I sort of liked Macro-10 ....
Simon Clubley
2017-01-10 18:48:03 UTC
Permalink
Post by David Froble
I sort of liked Macro-10 ....
I didn't realise you were a TOPS person. Which TOPS did you use ?

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
David Froble
2017-01-10 20:03:51 UTC
Permalink
Post by Simon Clubley
Post by David Froble
I sort of liked Macro-10 ....
I didn't realise you were a TOPS person. Which TOPS did you use ?
Simon.
It was a rather long time ago. University of Pittsburgh had PDP-10 systems.
TOPS-10. I was just a student, and when I got into some classes that used
assembler, I was a happy camper. Different than most of the people in the
class. Ended up doing much more than the class work, as I liked the language.
Guess I'm just a strange guy.

Never had a chance to use it after I graduated. Personal computers weren't back
then.
Simon Clubley
2017-01-10 20:44:57 UTC
Permalink
Post by David Froble
It was a rather long time ago. University of Pittsburgh had PDP-10 systems.
TOPS-10. I was just a student, and when I got into some classes that used
assembler, I was a happy camper. Different than most of the people in the
class. Ended up doing much more than the class work, as I liked the language.
Guess I'm just a strange guy.
Never had a chance to use it after I graduated. Personal computers weren't back
then.
I don't know if you are aware, but there are various free PDP-10
emulators available so there's nothing to stop you from running
it again if you wish.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
V***@SendSpamHere.ORG
2017-01-10 13:05:06 UTC
Permalink
Post by David Froble
Post by V***@SendSpamHere.ORG
Post by Bill Gunshannon
Post by c***@gmail.com
Post by Stephen Hoffman
Post by John Reagan
Hoff,
You skipped the one question I would have also considered interesting.
Yes, I did.
That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.
The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.
As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.
I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.
What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.
As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.
Post by John Reagan
"Any idea what language(s) were used to implement the DLM? "
I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)
If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.
Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...
There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.
The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.
VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.
I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)
In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.
https://www.fossil-scm.org
https://www.rust-lang.org
--
Pure Personal Opinion | HoffmanLabs LLC
Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%
These totals do NOT include the loader, layered products, compilers, DECnet V, TCPIP, and DECwindows.
Ada? Doing what? :-)
And, I thought MACRO was the VMS Assembler. Is there another?
bill
On Alpha and Itanium, Macro is compiled. Both Alpha and Itanium have native assemblers.
Ayep! And, if you got any smarts at all, you'll run, fast, screaming, from
either one.
I rather enjoyed the Alpha's assembly. I done Itanium too but there's more
"thought" involved to get it right.
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
V***@SendSpamHere.ORG
2017-01-09 23:51:37 UTC
Permalink
Post by Bill Gunshannon
Post by David Froble
Post by Stephen Hoffman
Post by Craig A. Berry
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC...
Works for me. (But then I'd like also to see a byte-addressable
file system, too.)
The existing byte-range locking and SSIO and C jackets is something
most of us would rather not have to deal with.
Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually
using interlocked queues for that sort of thing now, and that isn't
always the nicest approach for some of the designs.
An immediate question comes to mind. What is the magnitude of what
you're talking about?
"It depends". Anything from a month or so for a DLM-interlocked memory
design and implementation and testing assuming a small and
well-specified and single-host requirement, on up to scheduling
infinity. Where in this range depending on what is and is not in scope
in the project, performance requirements and related, and how compatible
the design is to any existing source code and available tools. It's a
how-high-is-up right now.
Post by David Froble
One question I've asked myself is, how big of integers might be needed
to define a range? Maybe quadword?
In OpenVMS it'd be a quadword, as that's the size of the addresses given
current virtual addressing limit of OpenVMS. There's a a LCK data
structure pointing to that range block. Based on the definitions, the
relevant data structure appears to be named LKIDEF_BR, though that
structure does not appear to have been exposed in the Bliss or C
definitions for V8.4.
Post by David Froble
Yes, what I'm considering would be perfect for what you mention, and
cluster aware, if that's a consideration.
There'll be overhead shuffling the data around; it's basically
server-level mirroring. Performance is going to come down to the
speed of the host-to-host, and that's going to be a whole lot of bespoke
code for networking traffic quite possibly involving some of the message
queue software and DECdtm. RDMA would be handy here too, certainly.
If the application requirements around performance permit it, I'd deploy
modern storage and 10 GbE networking and only cache the data in memory
when required. Might be able to brute-force the design and avoid
implementing mirrored memory, in other words. Flash and SSD are
massively faster than HDD, after all. That'd avoid a whole lot of
bespoke code, too. There's also the obvious and necessary discussion
around using one of the available commercial or open-source databases;
whether Oracle or otherwise.
Hoff,
You skipped the one question I would have also considered
interesting.
"Any idea what language(s) were used to implement the DLM? "
Macro
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
David Froble
2017-01-10 01:13:48 UTC
Permalink
Post by Bill Gunshannon
Post by David Froble
Post by Stephen Hoffman
Post by Craig A. Berry
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC...
Works for me. (But then I'd like also to see a byte-addressable
file system, too.)
The existing byte-range locking and SSIO and C jackets is something
most of us would rather not have to deal with.
Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually
using interlocked queues for that sort of thing now, and that isn't
always the nicest approach for some of the designs.
An immediate question comes to mind. What is the magnitude of what
you're talking about?
"It depends". Anything from a month or so for a DLM-interlocked memory
design and implementation and testing assuming a small and
well-specified and single-host requirement, on up to scheduling
infinity. Where in this range depending on what is and is not in scope
in the project, performance requirements and related, and how compatible
the design is to any existing source code and available tools. It's a
how-high-is-up right now.
Post by David Froble
One question I've asked myself is, how big of integers might be needed
to define a range? Maybe quadword?
In OpenVMS it'd be a quadword, as that's the size of the addresses given
current virtual addressing limit of OpenVMS. There's a a LCK data
structure pointing to that range block. Based on the definitions, the
relevant data structure appears to be named LKIDEF_BR, though that
structure does not appear to have been exposed in the Bliss or C
definitions for V8.4.
Post by David Froble
Yes, what I'm considering would be perfect for what you mention, and
cluster aware, if that's a consideration.
There'll be overhead shuffling the data around; it's basically
server-level mirroring. Performance is going to come down to the
speed of the host-to-host, and that's going to be a whole lot of bespoke
code for networking traffic quite possibly involving some of the message
queue software and DECdtm. RDMA would be handy here too, certainly.
If the application requirements around performance permit it, I'd deploy
modern storage and 10 GbE networking and only cache the data in memory
when required. Might be able to brute-force the design and avoid
implementing mirrored memory, in other words. Flash and SSD are
massively faster than HDD, after all. That'd avoid a whole lot of
bespoke code, too. There's also the obvious and necessary discussion
around using one of the available commercial or open-source databases;
whether Oracle or otherwise.
Hoff,
You skipped the one question I would have also considered
interesting.
"Any idea what language(s) were used to implement the DLM? "
Macro
Anyone remember Kirbey Altman?

"The final holocast, cause by a bug check" (or something like that)

And

"At least it won't come from Basic"

Fun times ....
David Froble
2017-01-07 01:59:19 UTC
Permalink
Post by Craig A. Berry
Post by David Froble
Post by Stephen Hoffman
Not sure how you're envisioning I/O would be connected to the DLM.
I'm not. The DLM should be sort of an object that does a job.
However, there is nothing that I'm aware of that does byte range
locking, or has that need, that's currently implemented. Perhaps I
should ignore the I/O part of it and just concentrate on doing locks
on byte ranges.
Byte-range locking is "just" a convention of resource names added within
and used (solely) within the C RTL. Some details of the resource names
used were posted (somewhere) a while back IIRC, but there's nothing
published about the internal implementation within the C RTL, AFAIK.
RMS and the XQP are where the file system and volume-level locking is
implemented respectively, and the C RTL then added some of its own
lock-based coordination specifically for the byte-range processing, as
well as the buffer shuffling entailed for that access as OpenVMS doesn't
have a byte-range I/O API. The caller — the C RTL in this case — has
to shuffle the data around within its own context. RMS I/O doesn't
know from C byte-range locking, it (RMS) has its own lock resource names
and its own conventions. C uses its own locks as well as having the
usual and typical dependencies of other RTL and application software on
OpenVMS around the coordination provided by lower-level software
including RMS and the XQP and the lock manager calls in those layers.
Somewhat related, there are some wrinkles here around what I/O
underneath C is atomic and what is not too, and that's what tripped up
the SSIO work.
What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC that bypasses all the stuff
described in your preceding paragraph.[1] It mystifies me why he would
want to start over from scratch rather than just encouraging VSI to fix
whatever kinks remain in SSIO and make it ready for prime time. He might
even advocate for the new API to be documented so it's available from
languages other than C :-).
[1] <http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf>
I'm not looking at starting over. I'm looking at an enhancement to the DLM.
The DLM is fully cluster aware and works fine in that environment. That is the
place to locate additional locking capabilities.

Maybe others think differently, but, I consider SSIO a language feature, and the
DLM an OS feature. ME, I'd choose the lower level for something that can be so
useful.

If what I'm considering is workable and is implemented, it would be something
that an SSIO implementation could use. And, other things could use also.
Thinking just SSIO is just too narrow of a focus, perhaps except for C bigots.
j***@yahoo.co.uk
2017-01-05 16:21:19 UTC
Permalink
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking for.
Can anyone with the books check in theirs and tell me which has DLM
information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.
I just don't have enough information now, and, will need some help, as right
now, I'd come up with a solution looking for a problem. For instance, Postgre
database, how large a range is reasonable, and if talking bytes, how many should
be supported, which would determine the size of integer required. I figure a
database could be rather large.
Don't know that I need exact information on the data structures. Whoever
implements such a modification would have to work from what's current in VMS.
But I need enough information to get close to what might be required.
One thing seems to be clear to me, it's not just the DLM, but I/O stuff would
have to also be implemented to use the new feature. There it gets interesting.
If using I/O buffers, how to determine the size for one thing. Or perhaps
direct I/O into the "on disk" file?
Post by Stephen Hoffman
Beyond the two manuals in the doc set and articles in the DEC and VMS
Technical Journals, Roy Davis' VAXcluster Principles book, most likely.
ISBN was 55558-112-9
Less so the IDSM. The DLM gets a fairly general discussion in the IDSM,
but not much in the way of details.
Or the OpenVMS source listings, for the full details.
Or for more general interest, skim through the OpenSSL open-source
variant of clustering software that Compaq released; that was a Unix DLM
similar to the OpenVMS DLM. Or skim the books and resources about one
of the more recent variants; see
https://en.wikipedia.org/wiki/Distributed_lock_manager for some pointers
to those.
If you do want to go for an IDSM for this or other reasons, you'll be
looking at a mix of V6 and the V7 updates to get all of the information
with the "latest" available information. The V6 stuff was the last
full copy of the IDSM, the V7 subset documents added chapters with some
details on specific and then-newer areas.
Closely related to the DLM is the distributed transaction manager;
DECdtm, and also the DEC RTR and the more recent open-source
0MQ/ActiveMQ/RabbitMQ messaging packages.
There is/was a Digital Technical Journal article on
the original architecture and design of the DLM.
It might be a bit, er, basic for your stated needs,
but you are in a better position than I am to make
that judgement.

It's in the VAXcluster Systems special edition of the
DTJ, from September 1987, pages 29-44.

I found it at
http://www.dtjcd.vmsresource.org.uk/pdfs/dtj_v01-05_sep1987.pdf
but other sources may also be available e.g. Bitsavers.

Are you aware that PATHWORKS did byte range locking
some twenty-odd years ago? Again, there is a DTJ special
issue with some potentially relevant info: Volume 4
Number 1, from 1992. The details have doubtless changed,
but the basics may or may not have changed.

Here are a few words from the piece on Byte Range Locking:
"Byte Range Locking
The MS-NET architecture allows for concurrent
access to server-based files by multiple clients. PC
applications acquire this functionality through the
MS-DOS byte range locking calls. These calls allow
PC applications to lock and unlock ranges of bytes
in a file and to detect conflicts. Conflicts occur
when part or all of a range specified to be locked
has been locked from a previous call. In contrast,
the approach taken by RMS provides locking on a
record basis. RMS uses the VMS distributed lock
manager to implement this functionality. Unfortu-
nately, the lock manager is not well suited to imple-
menting byte range locks because the byte range is
represented in a form that allows the lock manager
to arbitrate access. Therefore, the file server imple-
ments its own lock database and arbitrates access
to shared files. Internally, the server process main-
tains a list of locks for each file the server has open
and arbitrates access based on these lock struc-
tures. Files opened by the file server cannot be
shared with other VMS processes because the file
server has an exclusive mode lock on each file it has
open through the VMS lock manager. The exclusive
mode lock guarantees protection from other VMS
processes.
[etc]"


The DLM article abstract says:
The VMS distributed lock manager provides the
synchronization mechanism needed to ensure
transparent and reliable data sharing between
nodes in a VAXcluster system. The lock
manager provides services for mutual exclusion
and event notification, and achieves high
performance by minimizing the number of messages
sent between nodes. The lock manager also
handles deadlock situations with a minimum
of messages exchanged. Since processors systems
can join or leave a cluster at any time, a
connection manager was developed to handle
reconfigurations in a dynamic, efficient manner.
BillPedersen
2017-01-05 17:49:02 UTC
Permalink
Post by j***@yahoo.co.uk
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking for.
Can anyone with the books check in theirs and tell me which has DLM
information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.
I just don't have enough information now, and, will need some help, as right
now, I'd come up with a solution looking for a problem. For instance, Postgre
database, how large a range is reasonable, and if talking bytes, how many should
be supported, which would determine the size of integer required. I figure a
database could be rather large.
Don't know that I need exact information on the data structures. Whoever
implements such a modification would have to work from what's current in VMS.
But I need enough information to get close to what might be required.
One thing seems to be clear to me, it's not just the DLM, but I/O stuff would
have to also be implemented to use the new feature. There it gets interesting.
If using I/O buffers, how to determine the size for one thing. Or perhaps
direct I/O into the "on disk" file?
Post by Stephen Hoffman
Beyond the two manuals in the doc set and articles in the DEC and VMS
Technical Journals, Roy Davis' VAXcluster Principles book, most likely.
ISBN was 55558-112-9
Less so the IDSM. The DLM gets a fairly general discussion in the IDSM,
but not much in the way of details.
Or the OpenVMS source listings, for the full details.
Or for more general interest, skim through the OpenSSL open-source
variant of clustering software that Compaq released; that was a Unix DLM
similar to the OpenVMS DLM. Or skim the books and resources about one
of the more recent variants; see
https://en.wikipedia.org/wiki/Distributed_lock_manager for some pointers
to those.
If you do want to go for an IDSM for this or other reasons, you'll be
looking at a mix of V6 and the V7 updates to get all of the information
with the "latest" available information. The V6 stuff was the last
full copy of the IDSM, the V7 subset documents added chapters with some
details on specific and then-newer areas.
Closely related to the DLM is the distributed transaction manager;
DECdtm, and also the DEC RTR and the more recent open-source
0MQ/ActiveMQ/RabbitMQ messaging packages.
There is/was a Digital Technical Journal article on
the original architecture and design of the DLM.
It might be a bit, er, basic for your stated needs,
but you are in a better position than I am to make
that judgement.
It's in the VAXcluster Systems special edition of the
DTJ, from September 1987, pages 29-44.
I found it at
http://www.dtjcd.vmsresource.org.uk/pdfs/dtj_v01-05_sep1987.pdf
but other sources may also be available e.g. Bitsavers.
Are you aware that PATHWORKS did byte range locking
some twenty-odd years ago? Again, there is a DTJ special
issue with some potentially relevant info: Volume 4
Number 1, from 1992. The details have doubtless changed,
but the basics may or may not have changed.
"Byte Range Locking
The MS-NET architecture allows for concurrent
access to server-based files by multiple clients. PC
applications acquire this functionality through the
MS-DOS byte range locking calls. These calls allow
PC applications to lock and unlock ranges of bytes
in a file and to detect conflicts. Conflicts occur
when part or all of a range specified to be locked
has been locked from a previous call. In contrast,
the approach taken by RMS provides locking on a
record basis. RMS uses the VMS distributed lock
manager to implement this functionality. Unfortu-
nately, the lock manager is not well suited to imple-
menting byte range locks because the byte range is
represented in a form that allows the lock manager
to arbitrate access. Therefore, the file server imple-
ments its own lock database and arbitrates access
to shared files. Internally, the server process main-
tains a list of locks for each file the server has open
and arbitrates access based on these lock struc-
tures. Files opened by the file server cannot be
shared with other VMS processes because the file
server has an exclusive mode lock on each file it has
open through the VMS lock manager. The exclusive
mode lock guarantees protection from other VMS
processes.
[etc]"
The VMS distributed lock manager provides the
synchronization mechanism needed to ensure
transparent and reliable data sharing between
nodes in a VAXcluster system. The lock
manager provides services for mutual exclusion
and event notification, and achieves high
performance by minimizing the number of messages
sent between nodes. The lock manager also
handles deadlock situations with a minimum
of messages exchanged. Since processors systems
can join or leave a cluster at any time, a
connection manager was developed to handle
reconfigurations in a dynamic, efficient manner.
It should be noted that the Pathworks implementation works nicely for serving to PCs as it is a single process based implementation.

Samba/CIFS in a multi-process implementation as is PostgreSQL and so there is then the need to coordinate between processes as well as between cluster members. It would also be nice to be able to access the files from the VMS host...

It should also be noted that even Samba, and I expect PostgreSQL as well, implements its own byte range locking due to limitations in the fcntl() function call to allow access to LARGE files. So even there we have to consider some interesting issues. Samba implements an opportunistic locking mechanism which allows a client to have a cached, local copy of the file until such time as a second client comes along... This is not supported by VMS obviously. And is only supported by a few hosts as of now, from what I can tell. Samba also supports the deny-mode locks of Windows.

So, no matter what we do we will need to see how we can accommodate the locking in a way that eventually we can access from VMS as well as the clients.

Bill.
RobertsonEricW
2017-01-09 14:54:51 UTC
Permalink
Post by BillPedersen
Post by j***@yahoo.co.uk
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking for.
Can anyone with the books check in theirs and tell me which has DLM
information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.
I just don't have enough information now, and, will need some help, as right
now, I'd come up with a solution looking for a problem. For instance, Postgre
database, how large a range is reasonable, and if talking bytes, how many should
be supported, which would determine the size of integer required. I figure a
database could be rather large.
Don't know that I need exact information on the data structures. Whoever
implements such a modification would have to work from what's current in VMS.
But I need enough information to get close to what might be required.
One thing seems to be clear to me, it's not just the DLM, but I/O stuff would
have to also be implemented to use the new feature. There it gets interesting.
If using I/O buffers, how to determine the size for one thing. Or perhaps
direct I/O into the "on disk" file?
Post by Stephen Hoffman
Beyond the two manuals in the doc set and articles in the DEC and VMS
Technical Journals, Roy Davis' VAXcluster Principles book, most likely.
ISBN was 55558-112-9
Less so the IDSM. The DLM gets a fairly general discussion in the IDSM,
but not much in the way of details.
Or the OpenVMS source listings, for the full details.
Or for more general interest, skim through the OpenSSL open-source
variant of clustering software that Compaq released; that was a Unix DLM
similar to the OpenVMS DLM. Or skim the books and resources about one
of the more recent variants; see
https://en.wikipedia.org/wiki/Distributed_lock_manager for some pointers
to those.
If you do want to go for an IDSM for this or other reasons, you'll be
looking at a mix of V6 and the V7 updates to get all of the information
with the "latest" available information. The V6 stuff was the last
full copy of the IDSM, the V7 subset documents added chapters with some
details on specific and then-newer areas.
Closely related to the DLM is the distributed transaction manager;
DECdtm, and also the DEC RTR and the more recent open-source
0MQ/ActiveMQ/RabbitMQ messaging packages.
There is/was a Digital Technical Journal article on
the original architecture and design of the DLM.
It might be a bit, er, basic for your stated needs,
but you are in a better position than I am to make
that judgement.
It's in the VAXcluster Systems special edition of the
DTJ, from September 1987, pages 29-44.
I found it at
http://www.dtjcd.vmsresource.org.uk/pdfs/dtj_v01-05_sep1987.pdf
but other sources may also be available e.g. Bitsavers.
Are you aware that PATHWORKS did byte range locking
some twenty-odd years ago? Again, there is a DTJ special
issue with some potentially relevant info: Volume 4
Number 1, from 1992. The details have doubtless changed,
but the basics may or may not have changed.
"Byte Range Locking
The MS-NET architecture allows for concurrent
access to server-based files by multiple clients. PC
applications acquire this functionality through the
MS-DOS byte range locking calls. These calls allow
PC applications to lock and unlock ranges of bytes
in a file and to detect conflicts. Conflicts occur
when part or all of a range specified to be locked
has been locked from a previous call. In contrast,
the approach taken by RMS provides locking on a
record basis. RMS uses the VMS distributed lock
manager to implement this functionality. Unfortu-
nately, the lock manager is not well suited to imple-
menting byte range locks because the byte range is
represented in a form that allows the lock manager
to arbitrate access. Therefore, the file server imple-
ments its own lock database and arbitrates access
to shared files. Internally, the server process main-
tains a list of locks for each file the server has open
and arbitrates access based on these lock struc-
tures. Files opened by the file server cannot be
shared with other VMS processes because the file
server has an exclusive mode lock on each file it has
open through the VMS lock manager. The exclusive
mode lock guarantees protection from other VMS
processes.
[etc]"
The VMS distributed lock manager provides the
synchronization mechanism needed to ensure
transparent and reliable data sharing between
nodes in a VAXcluster system. The lock
manager provides services for mutual exclusion
and event notification, and achieves high
performance by minimizing the number of messages
sent between nodes. The lock manager also
handles deadlock situations with a minimum
of messages exchanged. Since processors systems
can join or leave a cluster at any time, a
connection manager was developed to handle
reconfigurations in a dynamic, efficient manner.
It should be noted that the Pathworks implementation works nicely for serving to PCs as it is a single process based implementation.
Samba/CIFS in a multi-process implementation as is PostgreSQL and so there is then the need to coordinate between processes as well as between cluster members. It would also be nice to be able to access the files from the VMS host...
It should also be noted that even Samba, and I expect PostgreSQL as well, implements its own byte range locking due to limitations in the fcntl() function call to allow access to LARGE files. So even there we have to consider some interesting issues. Samba implements an opportunistic locking mechanism which allows a client to have a cached, local copy of the file until such time as a second client comes along... This is not supported by VMS obviously. And is only supported by a few hosts as of now, from what I can tell. Samba also supports the deny-mode locks of Windows.
So, no matter what we do we will need to see how we can accommodate the locking in a way that eventually we can access from VMS as well as the clients.
Bill.
Do the existing (or newly imagined) Open Source uses of byte-range locking involve multi-threaded process contexts in which multiple, independent user-mode threads within the same process may attempt to access a particular byte range for a particular file?

The reason I ask is that the OpenVMS System Services locking API as represented by SYS$ENQ(W),SYS$DEQ, SYS$GETLKI, and SYS$SET_RESOURCE_DOMAIN grant, convert, snapshot, and destroy OpenVMS Resource Lock and Domain state at the process level. If multi-threading is potentially in play, then there will also need to be code written as part of this hypothetical byte-range locking API that can specifically deal with such multi-threading contexts so that multiple threads can independently (and transparently) manipulate Resource Locks and Domains (when necessary).

Apologies in advance if this complicates any hypothetical implementation. :)
David Froble
2017-01-09 15:54:40 UTC
Permalink
Post by RobertsonEricW
Do the existing (or newly imagined) Open Source uses of byte-range locking
involve multi-threaded process contexts in which multiple, independent
user-mode threads within the same process may attempt to access a particular
byte range for a particular file?
The reason I ask is that the OpenVMS System Services locking API as
represented by SYS$ENQ(W),SYS$DEQ, SYS$GETLKI, and SYS$SET_RESOURCE_DOMAIN
grant, convert, snapshot, and destroy OpenVMS Resource Lock and Domain state
at the process level. If multi-threading is potentially in play, then there
will also need to be code written as part of this hypothetical byte-range
locking API that can specifically deal with such multi-threading contexts so
that multiple threads can independently (and transparently) manipulate
Resource Locks and Domains (when necessary).
Apologies in advance if this complicates any hypothetical implementation. :)
I don't see how this can be a problem. When anything requests a lock which is
granted, it's totally up to the user of that lock to manage the lock. The lock
remains in the DLM database until it is released, or image rundown. The latter
case would be, in my opinion, defective code, so normally, the thread that has
the lock will release the lock when finished with it. Additional threads would
obtain their own locks, if required, for any operations. Multiple threads could
not assume the rights of a lock granted to another thread.

At least that's how I understand things ....
RobertsonEricW
2017-01-09 17:06:08 UTC
Permalink
Post by David Froble
Post by RobertsonEricW
Do the existing (or newly imagined) Open Source uses of byte-range locking
involve multi-threaded process contexts in which multiple, independent
user-mode threads within the same process may attempt to access a particular
byte range for a particular file?
The reason I ask is that the OpenVMS System Services locking API as
represented by SYS$ENQ(W),SYS$DEQ, SYS$GETLKI, and SYS$SET_RESOURCE_DOMAIN
grant, convert, snapshot, and destroy OpenVMS Resource Lock and Domain state
at the process level. If multi-threading is potentially in play, then there
will also need to be code written as part of this hypothetical byte-range
locking API that can specifically deal with such multi-threading contexts so
that multiple threads can independently (and transparently) manipulate
Resource Locks and Domains (when necessary).
Apologies in advance if this complicates any hypothetical implementation. :)
I don't see how this can be a problem. When anything requests a lock which is
granted, it's totally up to the user of that lock to manage the lock. The lock
remains in the DLM database until it is released, or image rundown. The latter
case would be, in my opinion, defective code, so normally, the thread that has
the lock will release the lock when finished with it. Additional threads would
obtain their own locks, if required, for any operations. Multiple threads could
not assume the rights of a lock granted to another thread.
At least that's how I understand things ....
If the existing uses of the byte-range locking API within the existing Open Source under consideration implicitly treats the locking requests on a process basis, then your response may be correct. Otherwise, you would be stuck with dealing with multiple threads whether you liked it or not. However, after having thought just a little bit about this, the only cases that get complicated with multi-threading involve transitions to or from Resource Lock states states of Protected Read, Protected Write, Concurrent Write, and Exclusive. All of these state transition problems could be dealt with using a per-Lock Resource Mutex associated with the Resource Lock request. Each thread would need to acquire this mutex before proceeding with the usual process level Resource Lock usage or state transition.
Stephen Hoffman
2017-01-09 22:58:39 UTC
Permalink
Post by David Froble
Post by RobertsonEricW
Do the existing (or newly imagined) Open Source uses of byte-range
locking involve multi-threaded process contexts in which multiple,
independent user-mode threads within the same process may attempt to
access a particular byte range for a particular file?
I don't see how this can be a problem.
You're not dealing with threads David, and BASIC has no concept of
threading. Not without using the KP-related system service calls to
explicitly implement that in your code. So not likely a problem for
your BASIC environment, if you're using system services and BASIC and
not implementing threading locally and not using C threading-related
calls...
Post by David Froble
When anything requests a lock which is granted, it's totally up to the
user of that lock to manage the lock.
Threads are routines executing within the context of one application,
and quite possibly fully in parallel. This processing is
differentiated from ASTs which are either active or not, interrupting
the mainline processing while active. The OpenVMS locking-related
system services should be fine with threading — should be, but I'm not
entirely certain anybody's reviewed and confirmed that — but the C
byte-range locking APIs are not necessarily thread safe, and the POSIX
locking APIs are quite possibly inherently broken around threading.
If your BASIC code calls C code with byte-range locks using the POSIX
APIs, things might get a little more "interesting", obviously. And
it's apparently currently only the byte-range locking calls made via
the C API that are supported and documented.
--
Pure Personal Opinion | HoffmanLabs LLC
David Froble
2017-01-09 23:11:06 UTC
Permalink
Post by Stephen Hoffman
Post by David Froble
Post by RobertsonEricW
Do the existing (or newly imagined) Open Source uses of byte-range
locking involve multi-threaded process contexts in which multiple,
independent user-mode threads within the same process may attempt to
access a particular byte range for a particular file?
I don't see how this can be a problem.
You're not dealing with threads David, and BASIC has no concept of
threading. Not without using the KP-related system service calls to
explicitly implement that in your code. So not likely a problem for
your BASIC environment, if you're using system services and BASIC and
not implementing threading locally and not using C threading-related
calls...
Post by David Froble
When anything requests a lock which is granted, it's totally up to the
user of that lock to manage the lock.
Threads are routines executing within the context of one application,
and quite possibly fully in parallel. This processing is
differentiated from ASTs which are either active or not, interrupting
the mainline processing while active. The OpenVMS locking-related
system services should be fine with threading — should be, but I'm not
entirely certain anybody's reviewed and confirmed that — but the C
byte-range locking APIs are not necessarily thread safe, and the POSIX
locking APIs are quite possibly inherently broken around threading. If
your BASIC code calls C code with byte-range locks using the POSIX APIs,
things might get a little more "interesting", obviously. And it's
apparently currently only the byte-range locking calls made via the C
API that are supported and documented.
I think it's past time to drop the term "byte range locking". That is a
particular need, not a good description of a capability.

Perhaps numeric range locking would be much more appropriate. For sure such a
capability could lend itself to more than just locking a range of bytes in a file.

I can imagine other "types" of locks, though I haven't imagined what they might
be, yet. And that's putting the cart miles ahead of the horse, and it's not
clear the horse is even heading in the same direction ....
Stephen Hoffman
2017-01-10 00:32:28 UTC
Permalink
Post by David Froble
I think it's past time to drop the term "byte range locking". That is
a particular need, not a good description of a capability.
Perhaps numeric range locking would be much more appropriate. For sure
such a capability could lend itself to more than just locking a range
of bytes in a file.
I can imagine other "types" of locks, though I haven't imagined what
they might be, yet. And that's putting the cart miles ahead of the
horse, and it's not clear the horse is even heading in the same
direction ....
The associated C locking calls — which is presently the only documented
part of this whole OpenVMS implementation — are called the byte-range
locking calls.

That's the nomenclature used in the applicable standards documentation
and the OpenVMS C documentation.

That's what everybody else on other platforms calls those routines and
that capability, too.
--
Pure Personal Opinion | HoffmanLabs LLC
David Froble
2017-01-10 00:48:50 UTC
Permalink
Post by Stephen Hoffman
Post by David Froble
Post by RobertsonEricW
Do the existing (or newly imagined) Open Source uses of byte-range
locking involve multi-threaded process contexts in which multiple,
independent user-mode threads within the same process may attempt to
access a particular byte range for a particular file?
I don't see how this can be a problem.
You're not dealing with threads David, and BASIC has no concept of
threading. Not without using the KP-related system service calls to
explicitly implement that in your code. So not likely a problem for
your BASIC environment, if you're using system services and BASIC and
not implementing threading locally and not using C threading-related
calls...
Post by David Froble
When anything requests a lock which is granted, it's totally up to the
user of that lock to manage the lock.
Threads are routines executing within the context of one application,
and quite possibly fully in parallel. This processing is
differentiated from ASTs which are either active or not, interrupting
the mainline processing while active. The OpenVMS locking-related
system services should be fine with threading — should be, but I'm not
entirely certain anybody's reviewed and confirmed that — but the C
byte-range locking APIs are not necessarily thread safe, and the POSIX
locking APIs are quite possibly inherently broken around threading. If
your BASIC code calls C code with byte-range locks using the POSIX APIs,
things might get a little more "interesting", obviously. And it's
apparently currently only the byte-range locking calls made via the C
API that are supported and documented.
Well, we all know I don't get out much, but at least:

While there is a one to one correspondence between a process and an executing
image (I think), there are still can be multiple processes running under VMS,
and they can all choose to cooperate. So, why cannot multiple threads choose to
cooperate?

You're correct, I don't use threads, and could be unaware of subtle things, but
that admitted, a cooperating process, or image, or thread, either cooperates, or
it doesn't, right?

I'll also admit that at this time, due to the little I've read about the
"special case" +(my terminology) BRL in the CRTL, I'm choosing to ignore it and
just look at the DLM, and how to add additional "types" of locks to what it
already does. Not sure how it will work out, if at all, but, to me it seems the
"right" way to approach the issue.
Stephen Hoffman
2017-01-10 01:15:20 UTC
Permalink
Post by David Froble
While there is a one to one correspondence between a process and an
executing image (I think),
A multithreaded application is treated as separate processes sharing
the same address space, the same variables, heap and the rest of the
context.
Post by David Froble
there are still can be multiple processes running under VMS, and they
can all choose to cooperate.
Ayup, at least as far as shared memory that's been mapped among the
processes, and files and other such resources.
Post by David Froble
So, why cannot multiple threads choose to cooperate?
Threads inherently cooperate, as they're in the same address space,
running the same executable code, using the same variables and the same
heap. In parallel.

Multiple separate processes — the traditional design — only explicitly
cooperates among the processes involved, and only shares what they've
explicitly mapped into memory across the processes or are accessing
file storage and such.

From BASIC, you'd have to use the KP threads calls to implement
multithreading; akin to the behavior within C applications when the
POSIX threading-related calls are used.
Post by David Froble
You're correct, I don't use threads, and could be unaware of subtle
things, but that admitted, a cooperating process, or image, or thread,
either cooperates, or it doesn't, right?
Yes, but the POSIX C calls — the API design itself — are somewhat of a
dog's breakfast around multi-threading.
Post by David Froble
I'll also admit that at this time, due to the little I've read about
the "special case" +(my terminology) BRL in the CRTL, I'm choosing to
ignore it and just look at the DLM, and how to add additional "types"
of locks to what it already does. Not sure how it will work out, if at
all, but, to me it seems the "right" way to approach the issue.
I'd expect the system services would be thread-safe, but — until that's
documented and supported — we're probably left using the thread-unsafe
POSIX C byte-range locking calls.
--
Pure Personal Opinion | HoffmanLabs LLC
Simon Clubley
2017-01-10 18:53:25 UTC
Permalink
$ set response/mode=good_natured

You know, if you keep saying that, I'm going to suggest you get out
for some nice day walks in England's green and pleasant lands (and
there's some very pleasant lands around here) while I educate you
on all the technologies you are missing out on. :-)

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
V***@SendSpamHere.ORG
2017-01-10 20:08:56 UTC
Permalink
Post by Simon Clubley
$ set response/mode=good_natured
You know, if you keep saying that, I'm going to suggest you get out
for some nice day walks in England's green and pleasant lands (and
there's some very pleasant lands around here) while I educate you
on all the technologies you are missing out on. :-)
And did those feet in ancient times,
Walk upon England's mountains green?
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
Simon Clubley
2017-01-10 20:45:58 UTC
Permalink
Post by V***@SendSpamHere.ORG
Post by Simon Clubley
You know, if you keep saying that, I'm going to suggest you get out
for some nice day walks in England's green and pleasant lands (and
there's some very pleasant lands around here) while I educate you
on all the technologies you are missing out on. :-)
And did those feet in ancient times,
Walk upon England's mountains green?
Congratulations. You pass the test. :-)

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
David Froble
2017-01-10 20:13:20 UTC
Permalink
Post by Simon Clubley
$ set response/mode=good_natured
You know, if you keep saying that, I'm going to suggest you get out
for some nice day walks in England's green and pleasant lands (and
there's some very pleasant lands around here) while I educate you
on all the technologies you are missing out on. :-)
Simon.
Well, I am waiting for some nice days, but to do some flying, not walking.

:-)
Stephen Hoffman
2017-01-09 17:55:31 UTC
Permalink
Post by RobertsonEricW
Do the existing (or newly imagined) Open Source uses of byte-range
locking involve multi-threaded process contexts in which multiple,
independent user-mode threads within the same process may attempt to
access a particular byte range for a particular file?
Supposition based on existing and documented capabilities and
behaviors: the DLM is and would remain thread-safe.

The POSIX locking APIs have issues around threading, though.

What the OpenVMS C RTL POSIX APIs might provide for thread safety? Donno.

But given the issues around the POSIX APIs and threading, I'm not hopeful.

https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html
--
Pure Personal Opinion | HoffmanLabs LLC
David Froble
2017-01-05 17:49:06 UTC
Permalink
Post by j***@yahoo.co.uk
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking for.
Can anyone with the books check in theirs and tell me which has DLM
information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.
I just don't have enough information now, and, will need some help, as right
now, I'd come up with a solution looking for a problem. For instance, Postgre
database, how large a range is reasonable, and if talking bytes, how many should
be supported, which would determine the size of integer required. I figure a
database could be rather large.
Don't know that I need exact information on the data structures. Whoever
implements such a modification would have to work from what's current in VMS.
But I need enough information to get close to what might be required.
One thing seems to be clear to me, it's not just the DLM, but I/O stuff would
have to also be implemented to use the new feature. There it gets interesting.
If using I/O buffers, how to determine the size for one thing. Or perhaps
direct I/O into the "on disk" file?
Post by Stephen Hoffman
Beyond the two manuals in the doc set and articles in the DEC and VMS
Technical Journals, Roy Davis' VAXcluster Principles book, most likely.
ISBN was 55558-112-9
Less so the IDSM. The DLM gets a fairly general discussion in the IDSM,
but not much in the way of details.
Or the OpenVMS source listings, for the full details.
Or for more general interest, skim through the OpenSSL open-source
variant of clustering software that Compaq released; that was a Unix DLM
similar to the OpenVMS DLM. Or skim the books and resources about one
of the more recent variants; see
https://en.wikipedia.org/wiki/Distributed_lock_manager for some pointers
to those.
If you do want to go for an IDSM for this or other reasons, you'll be
looking at a mix of V6 and the V7 updates to get all of the information
with the "latest" available information. The V6 stuff was the last
full copy of the IDSM, the V7 subset documents added chapters with some
details on specific and then-newer areas.
Closely related to the DLM is the distributed transaction manager;
DECdtm, and also the DEC RTR and the more recent open-source
0MQ/ActiveMQ/RabbitMQ messaging packages.
There is/was a Digital Technical Journal article on
the original architecture and design of the DLM.
It might be a bit, er, basic for your stated needs,
but you are in a better position than I am to make
that judgement.
It's in the VAXcluster Systems special edition of the
DTJ, from September 1987, pages 29-44.
Thanks John. Read that when it came out, and again recently. Nothing real
helpful there for what I'm looking at.
Post by j***@yahoo.co.uk
I found it at
http://www.dtjcd.vmsresource.org.uk/pdfs/dtj_v01-05_sep1987.pdf
but other sources may also be available e.g. Bitsavers.
Are you aware that PATHWORKS did byte range locking
some twenty-odd years ago? Again, there is a DTJ special
issue with some potentially relevant info: Volume 4
Number 1, from 1992. The details have doubtless changed,
but the basics may or may not have changed.
No, I wasn't aware of that. However, I'm thinking that it didn'tuse the DLM,
and I'm looking at an enhance to the DLM to provide a generic solution for such
needs.
Post by j***@yahoo.co.uk
"Byte Range Locking
The MS-NET architecture allows for concurrent
access to server-based files by multiple clients. PC
applications acquire this functionality through the
MS-DOS byte range locking calls. These calls allow
PC applications to lock and unlock ranges of bytes
in a file and to detect conflicts. Conflicts occur
when part or all of a range specified to be locked
has been locked from a previous call. In contrast,
the approach taken by RMS provides locking on a
record basis. RMS uses the VMS distributed lock
manager to implement this functionality. Unfortu-
nately, the lock manager is not well suited to imple-
menting byte range locks because the byte range is
represented in a form that allows the lock manager
to arbitrate access. Therefore, the file server imple-
ments its own lock database and arbitrates access
to shared files. Internally, the server process main-
tains a list of locks for each file the server has open
and arbitrates access based on these lock struc-
tures. Files opened by the file server cannot be
shared with other VMS processes because the file
server has an exclusive mode lock on each file it has
open through the VMS lock manager. The exclusive
mode lock guarantees protection from other VMS
processes.
[etc]"
The VMS distributed lock manager provides the
synchronization mechanism needed to ensure
transparent and reliable data sharing between
nodes in a VAXcluster system. The lock
manager provides services for mutual exclusion
and event notification, and achieves high
performance by minimizing the number of messages
sent between nodes. The lock manager also
handles deadlock situations with a minimum
of messages exchanged. Since processors systems
can join or leave a cluster at any time, a
connection manager was developed to handle
reconfigurations in a dynamic, efficient manner.
Which works very well, including in a cluster. I'm just looking at the
possibility of extending the capabilities.
Chris Scheers
2017-01-05 19:26:18 UTC
Permalink
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking
for. Can anyone with the books check in theirs and tell me which has
DLM information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design?
Implementing a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions,
if I'm able to do so, are to come up with a preliminary design for
additions to the VMS DLM to implement byte range locking, then
distribute it to those interested to comment upon, add comments and
suggestions, and present the result to VSI as a potential modification.
It would be up to VSI as to whether to include it in VMS.
The DLM does support byte range locking (or at least it used to). But
it is undocumented.

AFAIK, for more information you need to look at the sources.
--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.

Voice: 817-237-3360 Internet: ***@applied-synergy.com
Fax: 817-237-3074
Bob Gezelter
2017-01-05 21:37:04 UTC
Permalink
Post by Chris Scheers
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking
for. Can anyone with the books check in theirs and tell me which has
DLM information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design?
Implementing a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions,
if I'm able to do so, are to come up with a preliminary design for
additions to the VMS DLM to implement byte range locking, then
distribute it to those interested to comment upon, add comments and
suggestions, and present the result to VSI as a potential modification.
It would be up to VSI as to whether to include it in VMS.
The DLM does support byte range locking (or at least it used to). But
it is undocumented.
AFAIK, for more information you need to look at the sources.
--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.
Fax: 817-237-3074
Chris,

With all due respect, a point of architecture.

The DLM manages locks. Locks have limited attributes and names.

How the locks are named and used by other components (e.g., RMS) is what determines what locks mean (e.g., records, files, byte ranges).

For example, one could create a run-time library that manages golf carts, and use the DLM to coordinate allocations (admittedly silly, one would want to use a database in actuality). The DLM is none the wiser, all it manages are lock names and values. The import of those names is outside the scope of the DLM.

- Bob Gezelter, http://www.rlgsc.com
David Froble
2017-01-05 22:15:46 UTC
Permalink
Post by Bob Gezelter
Post by Chris Scheers
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking
for. Can anyone with the books check in theirs and tell me which has
DLM information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design?
Implementing a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions,
if I'm able to do so, are to come up with a preliminary design for
additions to the VMS DLM to implement byte range locking, then
distribute it to those interested to comment upon, add comments and
suggestions, and present the result to VSI as a potential modification.
It would be up to VSI as to whether to include it in VMS.
The DLM does support byte range locking (or at least it used to). But
it is undocumented.
AFAIK, for more information you need to look at the sources.
--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.
Fax: 817-237-3074
Chris,
With all due respect, a point of architecture.
The DLM manages locks. Locks have limited attributes and names.
How the locks are named and used by other components (e.g., RMS) is what determines what locks mean (e.g., records, files, byte ranges).
For example, one could create a run-time library that manages golf carts, and use the DLM to coordinate allocations (admittedly silly, one would want to use a database in actuality). The DLM is none the wiser, all it manages are lock names and values. The import of those names is outside the scope of the DLM.
- Bob Gezelter, http://www.rlgsc.com
Correct. The DLM offers locks on "resource names" for cooperating processes.
It actually does no locking of anything. It just allows a user to request a
lock on a resource name, and if granted, will not issue another lock on that
resource until the first is released. Either the second user waits for the lock
to be granted, or, determines that the resource cannot be accessed. In reality,
if it wished to screw up things, it could go ahead and access the resource
anyway. The key word is "cooperating".

Lots of uses. Cause a batch job to wait. Cause user applications to be
synchronized, and such.
Jan-Erik Soderholm
2017-01-05 22:54:11 UTC
Permalink
Post by David Froble
Post by Bob Gezelter
Post by Chris Scheers
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking
for. Can anyone with the books check in theirs and tell me which has
DLM information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design?
Implementing a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My
intentions, if I'm able to do so, are to come up with a preliminary
design for additions to the VMS DLM to implement byte range locking,
then distribute it to those interested to comment upon, add comments
and suggestions, and present the result to VSI as a potential
modification. It would be up to VSI as to whether to include it in VMS.
The DLM does support byte range locking (or at least it used to). But
it is undocumented.
AFAIK, for more information you need to look at the sources.
--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.
Fax: 817-237-3074
Chris,
With all due respect, a point of architecture.
The DLM manages locks. Locks have limited attributes and names.
How the locks are named and used by other components (e.g., RMS) is what
determines what locks mean (e.g., records, files, byte ranges).
For example, one could create a run-time library that manages golf carts,
and use the DLM to coordinate allocations (admittedly silly, one would
want to use a database in actuality). The DLM is none the wiser, all it
manages are lock names and values. The import of those names is outside
the scope of the DLM.
- Bob Gezelter, http://www.rlgsc.com
Correct. The DLM offers locks on "resource names" for cooperating
processes. It actually does no locking of anything. It just allows a user
to request a lock on a resource name, and if granted, will not issue
another lock on that resource until the first is released.
If we take a database (Rdb in this case) it uses DLM to coordinate
it's locking, but is from the application point of view, not only
"getting access" or "not getting access". Maybe this is only by
the way Rdb interprets the locks in the DLM (?).

CR Concurrent Read:
The current transaction explicitly locks every record it
reads. Other transactions may read from and write to
the same table.

CW Concurrent Write:
The current transaction explicitly locks every record it
reads or writes. Other transactions may read from and
write to the same table.

PR Protected Read:
The current transaction does not need to lock any
records it reads. Other transactions may read from, but
not write to, the same table.

PW Protected Write:
The current transaction does not need to lock any
records it reads, but it locks every record it modifies or
adds. Other transactions may read in CR mode.

EX Exclusive:
The current transaction does not need to lock any
record it reads or modifies because no other access to
the table is allowed. Other transactions must wait
until this transaction finishes before they can have any
access to the table.


Rdb also modifies the locks on-the-fly when needed such as locking
the whole table, 1000 pages, 100 pages, 10 pages and then, if needed,
individual table rows. This gives fewer locking operation in most
cases. The feature is called "Adjustable lock granularity (ALG)".

Anyway, this shows an environment where the locking is not only
of the "yes/no" type but much more complex.



Either the
Post by David Froble
second user waits for the lock to be granted, or, determines that the
resource cannot be accessed. In reality, if it wished to screw up things,
it could go ahead and access the resource anyway. The key word is
"cooperating".
Lots of uses. Cause a batch job to wait. Cause user applications to be
synchronized, and such.
Chris Scheers
2017-01-06 23:42:55 UTC
Permalink
Post by David Froble
Post by Bob Gezelter
Post by Chris Scheers
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the
DLM, data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking
for. Can anyone with the books check in theirs and tell me which
has DLM information.
Not exactly sure what sort of information you're looking for,
here. Use of the DLM on OpenVMS? Interested in the DLM design?
Implementing a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My
intentions, if I'm able to do so, are to come up with a preliminary
design for additions to the VMS DLM to implement byte range locking,
then distribute it to those interested to comment upon, add comments
and suggestions, and present the result to VSI as a potential
modification. It would be up to VSI as to whether to include it in
VMS.
The DLM does support byte range locking (or at least it used to).
But it is undocumented.
AFAIK, for more information you need to look at the sources.
Chris,
With all due respect, a point of architecture.
The DLM manages locks. Locks have limited attributes and names.
How the locks are named and used by other components (e.g., RMS) is
what determines what locks mean (e.g., records, files, byte ranges).
For example, one could create a run-time library that manages golf
carts, and use the DLM to coordinate allocations (admittedly silly,
one would want to use a database in actuality). The DLM is none the
wiser, all it manages are lock names and values. The import of those
names is outside the scope of the DLM.
- Bob Gezelter, http://www.rlgsc.com
Correct. The DLM offers locks on "resource names" for cooperating
processes. It actually does no locking of anything. It just allows a
user to request a lock on a resource name, and if granted, will not
issue another lock on that resource until the first is released. Either
the second user waits for the lock to be granted, or, determines that
the resource cannot be accessed. In reality, if it wished to screw up
things, it could go ahead and access the resource anyway. The key word
is "cooperating".
Lots of uses. Cause a batch job to wait. Cause user applications to be
synchronized, and such.
OK, I misstated and you misunderstood.

Yes, I fully understand that the DLM does not enforce anything and use
of it is cooperative.

Granted, the DLM does not actually lock anything. What we are
discussing are resources.

In the "normal" usage of the DLM, a resource is a singular item. You
can lock "X", but that has no effect on a lock on "Y".

What you misunderstood is that the DLM already has the (undocumented)
capability for a resource to be a range. So if you lock the range
1000..2000, and then ask for a lock on 500..1500, a conflict should be
detected.

YMMV: I have not actually tried this as I have not had a need so far.
--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.

Voice: 817-237-3360 Internet: ***@applied-synergy.com
Fax: 817-237-3074
David Froble
2017-01-07 02:06:53 UTC
Permalink
Post by Chris Scheers
Post by David Froble
Post by Bob Gezelter
Post by Chris Scheers
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals
and such publications. What I'm looking for is information on
the DLM, data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking
for. Can anyone with the books check in theirs and tell me which
has DLM information.
Not exactly sure what sort of information you're looking for,
here. Use of the DLM on OpenVMS? Interested in the DLM design?
Implementing a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My
intentions, if I'm able to do so, are to come up with a preliminary
design for additions to the VMS DLM to implement byte range
locking, then distribute it to those interested to comment upon,
add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.
The DLM does support byte range locking (or at least it used to).
But it is undocumented.
AFAIK, for more information you need to look at the sources.
Chris,
With all due respect, a point of architecture.
The DLM manages locks. Locks have limited attributes and names.
How the locks are named and used by other components (e.g., RMS) is
what determines what locks mean (e.g., records, files, byte ranges).
For example, one could create a run-time library that manages golf
carts, and use the DLM to coordinate allocations (admittedly silly,
one would want to use a database in actuality). The DLM is none the
wiser, all it manages are lock names and values. The import of those
names is outside the scope of the DLM.
- Bob Gezelter, http://www.rlgsc.com
Correct. The DLM offers locks on "resource names" for cooperating
processes. It actually does no locking of anything. It just allows a
user to request a lock on a resource name, and if granted, will not
issue another lock on that resource until the first is released.
Either the second user waits for the lock to be granted, or,
determines that the resource cannot be accessed. In reality, if it
wished to screw up things, it could go ahead and access the resource
anyway. The key word is "cooperating".
Lots of uses. Cause a batch job to wait. Cause user applications to
be synchronized, and such.
OK, I misstated and you misunderstood.
Yes, I fully understand that the DLM does not enforce anything and use
of it is cooperative.
Granted, the DLM does not actually lock anything. What we are
discussing are resources.
In the "normal" usage of the DLM, a resource is a singular item. You
can lock "X", but that has no effect on a lock on "Y".
What you misunderstood is that the DLM already has the (undocumented)
capability for a resource to be a range. So if you lock the range
1000..2000, and then ask for a lock on 500..1500, a conflict should be
detected.
YMMV: I have not actually tried this as I have not had a need so far.
Well, if that is there, then job may be already completed. But I'm not aware of
such a capability. It requires the DLM to work on more than just resource
names, as you suggest, "X" and "Y".

You got any pointers to any information on such a capability?
Chris Scheers
2017-01-07 08:11:09 UTC
Permalink
Post by David Froble
Post by Chris Scheers
Post by David Froble
Post by Bob Gezelter
Post by Chris Scheers
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals
and such publications. What I'm looking for is information on
the DLM, data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking
for. Can anyone with the books check in theirs and tell me
which has DLM information.
Not exactly sure what sort of information you're looking for,
here. Use of the DLM on OpenVMS? Interested in the DLM
design? Implementing a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My
intentions, if I'm able to do so, are to come up with a
preliminary design for additions to the VMS DLM to implement byte
range locking, then distribute it to those interested to comment
upon, add comments and suggestions, and present the result to VSI
as a potential modification. It would be up to VSI as to whether
to include it in VMS.
The DLM does support byte range locking (or at least it used to).
But it is undocumented.
AFAIK, for more information you need to look at the sources.
Chris,
With all due respect, a point of architecture.
The DLM manages locks. Locks have limited attributes and names.
How the locks are named and used by other components (e.g., RMS) is
what determines what locks mean (e.g., records, files, byte ranges).
For example, one could create a run-time library that manages golf
carts, and use the DLM to coordinate allocations (admittedly silly,
one would want to use a database in actuality). The DLM is none the
wiser, all it manages are lock names and values. The import of those
names is outside the scope of the DLM.
- Bob Gezelter, http://www.rlgsc.com
Correct. The DLM offers locks on "resource names" for cooperating
processes. It actually does no locking of anything. It just allows a
user to request a lock on a resource name, and if granted, will not
issue another lock on that resource until the first is released.
Either the second user waits for the lock to be granted, or,
determines that the resource cannot be accessed. In reality, if it
wished to screw up things, it could go ahead and access the resource
anyway. The key word is "cooperating".
Lots of uses. Cause a batch job to wait. Cause user applications to
be synchronized, and such.
OK, I misstated and you misunderstood.
Yes, I fully understand that the DLM does not enforce anything and use
of it is cooperative.
Granted, the DLM does not actually lock anything. What we are
discussing are resources.
In the "normal" usage of the DLM, a resource is a singular item. You
can lock "X", but that has no effect on a lock on "Y".
What you misunderstood is that the DLM already has the (undocumented)
capability for a resource to be a range. So if you lock the range
1000..2000, and then ask for a lock on 500..1500, a conflict should be
detected.
YMMV: I have not actually tried this as I have not had a need so far.
Well, if that is there, then job may be already completed. But I'm not
aware of such a capability. It requires the DLM to work on more than
just resource names, as you suggest, "X" and "Y".
You got any pointers to any information on such a capability?
As I said, AFAIK, you need to look at the VMS sources for information.
--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.

Voice: 817-237-3360 Internet: ***@applied-synergy.com
Fax: 817-237-3074
John Reagan
2017-01-07 14:29:05 UTC
Permalink
I've stayed away from this all week, but I'll toss in a few facts.

Please, please, please don't call the range operand on $ENQ as byte-range locking. As many other notes have already said, the DLM doesn't "lock" anything. It is up to the cooperating processes to make it all work.

The CRTL's BRL is in there now via the fcntl() interface.

There essentially creates a bunch of DLM locks with the name C$_deviceAAAABBBBCCCC where ABC is the hex representation of the FID.

It uses sublocks to keep track of the ranges of bytes is it interested in.

The scheme uses various blocking ASTs so each process gets notified of another process accessing the range or attempting to also lock the range (or part of the range).

The range feature of the DLM simply allows some blocking asts to be skipped since the CRTL would simply decide that the ranges don't overlap and that there is nothing to do.

The code in the CRTL has to worry about overlapping ranges, combining adjacent ranges, splitting large ranges into smaller ones, removing system locks at image rundown, etc.

The code in the CRTL is split between DECC$SHR and DECC$SHRP since the actual locking uses SYSTEM wide locks.

From reading the comments, the routines DECC$$BRL_ENQ, DECC$$BRL_ENQW, DECC$$BRL_DEQ, and DECC$$BRL_GETLKIW were meant to be user-visible. There are various probes and parameter checking. I'm not sure if they were ever described in public and I'm not sure of what sort of application would use them directly.

To push BRL down to RMS or the XQP to provide the functionality to non-C applications, it would have to worry about all the things the CRTL does. It would have to worry about more things like what does byte-range locking mean for non-sequential files? I assume you wouldn't allow that since you can already lock records on indexed files. Do you want BRL on individual fields inside an indexed record?

I counted close to 7000 lines of code across both images.
David Froble
2017-01-07 20:22:29 UTC
Permalink
Post by John Reagan
I've stayed away from this all week, but I'll toss in a few facts.
Gee, I wish you would have done that tossing a month ago, I might have got more
work done in the shop. But, you had no idea of what I was considering ....
Post by John Reagan
Please, please, please don't call the range operand on $ENQ as byte-range
locking. As many other notes have already said, the DLM doesn't "lock"
anything. It is up to the cooperating processes to make it all work.
I agree 100%, but, the term "lock" has been in use since the mid 1980s. I think
we're stuck with it.
Post by John Reagan
The CRTL's BRL is in there now via the fcntl() interface.
There essentially creates a bunch of DLM locks with the name
C$_deviceAAAABBBBCCCC where ABC is the hex representation of the FID.
It uses sublocks to keep track of the ranges of bytes is it interested in.
Just as sublocks are used to specify other items of interest, such as record
numbers.
Post by John Reagan
The scheme uses various blocking ASTs so each process gets notified of
another process accessing the range or attempting to also lock the range (or
part of the range).
The range feature of the DLM simply allows some blocking asts to be skipped
since the CRTL would simply decide that the ranges don't overlap and that
there is nothing to do.
Just like RMS records, and other things.
Post by John Reagan
The code in the CRTL has to worry about overlapping ranges, combining
adjacent ranges, splitting large ranges into smaller ones, removing system
locks at image rundown, etc.
That I don't understand. The DLM has always (as far as I know) removed locks on
image rundown. Then again, you specified CRTL, not DLM.
Post by John Reagan
The code in the CRTL is split between DECC$SHR and DECC$SHRP since the actual
locking uses SYSTEM wide locks.
Not sure what good a non-system wide lock is when multiple processes are
accessing the resources in question.
Post by John Reagan
From reading the comments, the routines DECC$$BRL_ENQ, DECC$$BRL_ENQW,
DECC$$BRL_DEQ, and DECC$$BRL_GETLKIW were meant to be user-visible. There
are various probes and parameter checking. I'm not sure if they were ever
described in public and I'm not sure of what sort of application would use
them directly.
To push BRL down to RMS or the XQP to provide the functionality to non-C
applications, it would have to worry about all the things the CRTL does. It
would have to worry about more things like what does byte-range locking mean
for non-sequential files? I assume you wouldn't allow that since you can
already lock records on indexed files. Do you want BRL on individual fields
inside an indexed record?
I counted close to 7000 lines of code across both images.
This is beginning to sound like other things that happened when VMS was taken
away from competent developers. But I don't know when all this was done.
Regardless, I don't see what business the CRTL has in the entire scheme. Sure
sounds like someone intended a "patch" for a particular and limited need rather
than a well architectured enhanced capability.

But, thanks for the info ....
Craig A. Berry
2017-01-07 22:01:44 UTC
Permalink
Post by David Froble
Post by John Reagan
Please, please, please don't call the range operand on $ENQ as byte-range
locking. As many other notes have already said, the DLM doesn't "lock"
anything. It is up to the cooperating processes to make it all work.
The code in the CRTL has to worry about overlapping ranges, combining
adjacent ranges, splitting large ranges into smaller ones, removing
system locks at image rundown, etc.
That I don't understand. The DLM has always (as far as I know) removed
locks on image rundown. Then again, you specified CRTL, not DLM.
I don't think you were listening when he said the DLM doesn't lock
anything. Something else still has to do the actual synchronization of
I/O, and whatever the DLM does at image run-down is not going to undo
any of that. I suspect this was all implemented in the CRTL because
nothing else provided it, but in principle SSIO, if it works out, should
be a step forward by making it more general and more performant.
Craig A. Berry
2017-01-07 22:11:03 UTC
Permalink
[fcntl] essentially creates a bunch of DLM locks with the name
C$_deviceAAAABBBBCCCC where ABC is the hex representation of the FID.
Which may need revision when the components of the FID become 64-bit,
and as far as I can see the resource name argument to $ENQ is limited to
31 bytes. :-(
Stephen Hoffman
2017-01-07 22:42:51 UTC
Permalink
Post by John Reagan
I've stayed away from this all week, but I'll toss in a few facts.
Please, please, please don't call the range operand on $ENQ as
byte-range locking....
I think you're going to lose that one, at least prior to a stable API
and published documentation becoming available.
Post by John Reagan
As many other notes have already said, the DLM doesn't "lock"
anything. It is up to the cooperating processes to make it all work.
Ayup, and the L in DLM is for limerick, right? That ship has sailed.
Post by John Reagan
The CRTL's BRL is in there now via the fcntl() interface.
There essentially creates a bunch of DLM locks with the name
C$_deviceAAAABBBBCCCC where ABC is the hex representation of the FID.
It uses sublocks to keep track of the ranges of bytes is it interested in.
The scheme uses various blocking ASTs so each process gets notified of
another process accessing the range or attempting to also lock the
range (or part of the range).
The range feature of the DLM simply allows some blocking asts to be
skipped since the CRTL would simply decide that the ranges don't
overlap and that there is nothing to do.
The code in the CRTL has to worry about overlapping ranges, combining
adjacent ranges, splitting large ranges into smaller ones, removing
system locks at image rundown, etc.
The code in the CRTL is split between DECC$SHR and DECC$SHRP since the
actual locking uses SYSTEM wide locks.
From reading the comments, the routines DECC$$BRL_ENQ, DECC$$BRL_ENQW,
DECC$$BRL_DEQ, and DECC$$BRL_GETLKIW were meant to be user-visible.
There are various probes and parameter checking. I'm not sure if they
were ever described in public and I'm not sure of what sort of
application would use them directly.
They're not documented anywhere the search engines can locate. The $$
naming usually indicates a private symbol and not a public one, too.
Post by John Reagan
To push BRL down to RMS or the XQP to provide the functionality to
non-C applications, it would have to worry about all the things the
CRTL does. It would have to worry about more things like what does
byte-range locking mean for non-sequential files? I assume you
wouldn't allow that since you can already lock records on indexed
files. Do you want BRL on individual fields inside an indexed record?
I counted close to 7000 lines of code across both images.
For what I suspect David was seeking — using this to coordinate ranges
of memory, and not ranges wthin files — this project will then either
involve somehow recreating the C RTL portion of the coordinating code
in the app and then call the Distributed Limerick, err, Lock Manager,
or it'll involve the creation of a private lock manager. Having this
scattered between the DLM and the C RTL was probably expedient, but
AFAICT it does not seem particularly reusable.
--
Pure Personal Opinion | HoffmanLabs LLC
Bob Koehler
2017-01-06 14:59:04 UTC
Permalink
How the locks are named and used by other components (e.g., RMS) is what de=
termines what locks mean (e.g., records, files, byte ranges).
In particular, how RMS and XQP use DLM is not what one might naively
expect.
d***@gmail.com
2017-01-05 22:30:19 UTC
Permalink
Post by Chris Scheers
The DLM does support byte range locking (or at least it used to). But
it is undocumented.
AFAIK, for more information you need to look at the sources.
.SBTTL EXE$ENQ - Enqueue system service

;++
; FUNCTIONAL DESCRIPTION:
;
; This routine handles the $ENQ system service.
;
; CALLING SEQUENCE:
;
; CALLS/G EXE$ENQ (actually called through the system service
; dispatcher)
;
; INPUT PARAMETERS:
;
; ENQ_EFN(AP) Event flag number
; ENQ_LKMODE(AP) Lock mode
; ENQ_LKSB(AP) Address of lock status block
; ENQ_FLAGS(AP) Flags
; ENQ_RESNAM(AP) Address of descriptor of resource name
; ENQ_PARID(AP) Parent lock id
; ENQ_ASTADR(AP) Address of completion AST routine
; ENQ_ASTPRM(AP) AST parameter
; ENQ_BLKAST(AP) Address of blocking AST routine
; ENQ_ACMODE(AP) Access mode
; ENQ_RSDM_ID(AP) Resource Domain ID
; ENQ_RNGBLK(AP) Range Block
; ENQ_PRIORITY(AP) Priority of request
V***@SendSpamHere.ORG
2017-01-06 12:58:08 UTC
Permalink
Post by Chris Scheers
Post by David Froble
Post by Stephen Hoffman
Post by David Froble
I'm looking at a half dozen books on Amazon, all VMS internals and
such publications. What I'm looking for is information on the DLM,
data structures and design.
There is a V5.2 from 1987. Not sure if this is what I'm looking
for. Can anyone with the books check in theirs and tell me which has
DLM information.
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design?
Implementing a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions,
if I'm able to do so, are to come up with a preliminary design for
additions to the VMS DLM to implement byte range locking, then
distribute it to those interested to comment upon, add comments and
suggestions, and present the result to VSI as a potential modification.
It would be up to VSI as to whether to include it in VMS.
The DLM does support byte range locking (or at least it used to). But
it is undocumented.
Huh?
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
Simon Clubley
2017-01-06 18:39:59 UTC
Permalink
Post by David Froble
Post by Stephen Hoffman
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.
I haven't read the original Gray paper in a _long_ time.

Was there anything in there which might help ?

It seems to me that you are looking more for design papers which
discuss byte range locking than you are detailed implementation
notes of the existing DLM solution.

Have you tried looking further afield for how other locking systems
(distributed or otherwise) handle this ?

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
David Froble
2017-01-06 21:11:38 UTC
Permalink
Post by Simon Clubley
Post by David Froble
Post by Stephen Hoffman
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.
I haven't read the original Gray paper in a _long_ time.
Was there anything in there which might help ?
It seems to me that you are looking more for design papers which
discuss byte range locking than you are detailed implementation
notes of the existing DLM solution.
Have you tried looking further afield for how other locking systems
(distributed or otherwise) handle this ?
Simon.
There are a few things about the VMS DLM that need to be understood.

The first, and biggest item, is that the name is misleading. The DLM does NO
LOCKING. It doesn't have that capability.

The VMS DLM is cluster aware, and lots of work has been done to cause it to work
well in a VMS Cluster environment.

I'm using the term "lock" because it's in general use. It might be more
confusing to attempt to use another term.

What the VMS DLM currently does is accept requests for "locks", determine
whether to allow the request, and create an entry in it's database if the
request is allowed. Trying to keep this simple, no sense in getting into
pending locks, null locks, and such. Not really relavent.

The key issue is that a "lock" is granted for a "resource name". Doesn't matter
what the name is, could be "XYZZY", "Dave's a fool", anything. Subsequent
requests for locks on the identical name are considered upon what's already in
the DLM database. Critical key concept, the DLM doesn't care what the name is,
and does nothing, no evaluation, and such with the "resource name".

I know of implementations that take out a separate lock on each block when
attempting to lock multiple blocks in a file. I don't think it's practical to
take out a lock for every byte in a byte range. So, additional capability is
required.

I've been thinking about this for over a month. My best, and only, idea is that
to address byte range locking, and perhaps other types, is for the DLM to be
able to use data supplied with the lock request to determine whether to grant
the lock, and to store such data in all locks of this type. Since "class" is
already in use for some things, and I'm not really sure what those things are,
while I'd like to call it a "class of lock", maybe I should not do that. So
I'll just call it a "type" of lock.

For this, at least two pieces of data are required. First, an indicator of the
"type" of lock being requested. Second, the actual data to use to describe the
"resource" for which a lock is requested.

Type of lock would be arbitrary and up to the implimenter(s). It would perhaps
be another input parameter to be presented to the DLM. This means the current
type of locks would perhaps need to be assigned a "type". It would also most
likely be the "default type".

For range locking, the first and last element of the range would need to be
submitted to the DLM. This would be for the sub locks. Then there would need
to be new code, implemented by type of lock, to do evaluation of the data.
Note, this is a new concept for the DLM.

When looking for conflicts, the new code would need to search for conflicts in
the range. If found, deny or hold the request. If not found, grant the
request, which includes another entry in the DLM database.

I'm thinking the comparison of ranges will require some interesting concepts and
code. Haven't really thought about that. Got to be doable.

Right now I'm looking for documentation that might allow me to do some grunt
work for whoever might be assigned to implement the proposed enhancement.
Current DLM database table design and such. Hopefully enough to cause those
making decisions to consider the proposed enhancement.

My hope is that if the community, that's us, can present VSI with a concept and
plan that they feel they can implement, then perhaps we'd get the enhancement.

You up for it?
Simon Clubley
2017-01-09 00:34:47 UTC
Permalink
Post by David Froble
Post by Simon Clubley
Post by David Froble
Post by Stephen Hoffman
Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...
Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.
I haven't read the original Gray paper in a _long_ time.
Was there anything in there which might help ?
It seems to me that you are looking more for design papers which
discuss byte range locking than you are detailed implementation
notes of the existing DLM solution.
Have you tried looking further afield for how other locking systems
(distributed or otherwise) handle this ?
Simon.
There are a few things about the VMS DLM that need to be understood.
The first, and biggest item, is that the name is misleading. The DLM does NO
LOCKING. It doesn't have that capability.
[snip]

I know all this, but as Stephen has also pointed out, it was called
the Distributed _Lock_ Manager and not the Distributed _Resource_
Manager when it was designed back in the 1980s.

Blame the DEC people who choose the name, not those of us who just use
the name that DEC gave it. :-)
Post by David Froble
I'm thinking the comparison of ranges will require some interesting concepts and
code. Haven't really thought about that. Got to be doable.
Why ? If it's too difficult to prove the ranges overlap, you can
reverse the check by proving the ranges _don't_ overlap instead
of trying to prove they _do_ overlap.

In some cases, proving something doesn't overlap can be easier then
trying to prove it does.

Besides, AFAICS, it's just the normal intersection of sets that are
at play here.
Post by David Froble
Right now I'm looking for documentation that might allow me to do some grunt
work for whoever might be assigned to implement the proposed enhancement.
Current DLM database table design and such. Hopefully enough to cause those
making decisions to consider the proposed enhancement.
My hope is that if the community, that's us, can present VSI with a concept and
plan that they feel they can implement, then perhaps we'd get the enhancement.
You up for it?
Sorry but I'm currently working on variety of VMS related and
non-VMS projects in my spare time.

Now that you know the DLM supports a generic range locking capability,
I think you should instead now make a formal request to VSI to have
this formally documented (and completed if required).

Send off an email to Clair at the RnD VSI address with a formal
request (including your justification for why the documentation/work
is required) and see what he has to say.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world
Michael Moroney
2017-01-06 21:57:12 UTC
Permalink
Byte range locking appears to be in the DLM, but undocumented.

Are you looking for other internal/undocumented stuff for whatever
you are doing, or just byte range locking.

Perhaps your request should be for VSI to document byte range
locking after cleaning it up, if necessary.
David Froble
2017-01-07 02:14:22 UTC
Permalink
Post by Michael Moroney
Byte range locking appears to be in the DLM, but undocumented.
Are you looking for other internal/undocumented stuff for whatever
you are doing, or just byte range locking.
Perhaps your request should be for VSI to document byte range
locking after cleaning it up, if necessary.
Well, that's sure a teaser.

Is something implemented?

If so, is there any information about how it's implemented, it's capabilities,
and such.

I'd rather not go spinning my wheels trying to work up some specs on something
that already exists.
Stephen Hoffman
2017-01-07 04:17:13 UTC
Permalink
Post by David Froble
Is something implemented?
If so, is there any information about how it's implemented, it's
capabilities, and such.
C is using the API, and it's undocumented, which means it's either off
to the source listings for details. Or a discussion directly with
VSI. Or both.
--
Pure Personal Opinion | HoffmanLabs LLC
Loading...