David Barbour
2013-09-20 05:35:19 UTC
Over the last month, I feel like I stumbled into something very simple and
profound: a new perspective on an old idea, with consequences deeper and
more pervasive than I had imagined.
The idea is simply this: every user action is an act of meta-programming.
More precisely:
(1) Each user event addends a tacit concatenative program.
(2) The output of the tacit concatenative program is another program.
(3) We can understand the former as rewriting parts of the latter.
(4) These rewrites include the user-model - navigation, clipboard, etc.
I will further explain this idea, why it is powerful, how it is different.
To clarify, this isn't another hand-wavy 'shalt' and 'must' proposal with
no idea of how to achieve it. Hammering at a huge list of requirements for
eight years got me to RDP. At this point, I have concrete ideas on how to
accomplish everything I'm about to describe.
Users Are Programmers.
The TUNES vision is revived, and better than ever.
*WHY TACIT CONCATENATIVE?*
Concatenative programming is perhaps best known through FORTH. Most
concatenative languages have followed in Charles Moore's forthsteps,
sticking with the basic stack concept but focusing on higher-order
programming, types, and other features.
A stack would be an extremely impoverished and cramped environment for a
user; even many programmers would not tolerate it. Fortunately, we can move
beyond the stack environment. And I insist that we do! Concatenative
programming can also be based upon such structures as trees, Huet zippers,
and graphs. This proposal is based primarily on tree-structured data and
zippers, with just a little indirect graph modeling through shared state or
explicit labels (details later).
A 'tacit' programming language is one that does not mention names for
parameters or local variables. Many concatenative programming languages are
also tacit, though the concepts don't fully intersect.
A weakness of tacit concatenative programming is that, in a traditional
text-based programming environment, users must visualize the environment
(stack or other structure) in their head, and that they must memorize a
bunch of arcane 'stack shuffling' words. By comparison, variable names in
text are easy to visualize and review.
My answer: change programming environments!
Powerful advantages of tacit concatenative programming include:
1. the environment has a precisely defined, visualizable value
2. short strings of tacit concatenative code are easy to generate
3. concatenative code is sequential, forming an implicit timeline
4. code also subject to learning, pattern recognition, and rewrites
5. every step, small and large, is precisely defined and typed
Instead of an impoverished, text-based programming environment, we should
offer continuous automatic visualization. Rather than asking users to
memorize arcane words, we should offer direct manipulation: e.g. take, put,
slide, toss, drag, drop, copy, paste. Appropriate tacit concatenative code
is generated at every step, for every gesture. This code is easy to
generate because generators can focus on short vectors of learned 'known
useful' words without syntactic noise; this is subject to a variety of
well-known solutions (logical searches, genetic programming,
hill-climbing).
And then there are benefits that move beyond anything offered today, UI or
PL.
Not only can we visualize the environment, we can animate it. Users can
review and replay their actions, potentially from different perspectives or
highlighting different objects. Since even the smallest dataflow steps are
well defined, users can review at different temporal scales, based on the
granularity of their actions - zooming in to see precisely what taking an
object entails, or zooming out to see broad changes in an environment.
Rewrites can be used to make these animations smoother, more efficient, and
perhaps more aesthetically pleasing. And, in addition to undo, users can
rewrite parts of their history to better understand a decision or to fix a
mistake.
The programming environment can also help users construct macros: pattern
recognition is easy with tacit programming even if it were just in terms of
sequences of words. However, patterns are augmented further by looking at
context, the environment at the time a word was used. Proposed words can be
refined with very simple decision procedures to account for slight
context-sensitive variations. Discovered patterns can be used for simple
compression of history, or be used for programming-by-example.
An environment that recognizes a pattern might quietly and unobtrusively
offer a constructed tool or macro, that the user might refine a little
(e.g. clarifying the decision procedure) before using. The notion of
'dialecting' and 'DSLs' is replaced by problem-specific toolboxes and
macros, where a tool may act a lot like a paintbrush.
Further, there are advantages from the formalism and typing!
For one example, it is to guide user actions relevant to the typeful
context - i.e. making appropriate suggestions. Also, multiple actions can
be assigned to a single gesture or voice command, so long as they are
distinguishable in most typeful contexts. (When there seems to be
ambiguity, the environment can ask for clarification. Not a problem so long
as it's rare.)
By introspecting the environment, we can also create words that are 'smart'
about their application, i.e. automatically performing a search of the
local environment to find an appropriate target, and perhaps validate that
it is a unique target. This ability to be selectively imprecise can greatly
reduce the burden on users and developers. (Usefully, we can separate the
'search' and 'apply' patterns such that augmenting any action with search
is a simple composition.)
Tacit concatenative programming is *safer* than names.
With parameter based programming, the data-plumbing is untyped and ad-hoc.
Further, captured names are almost never visible in the 'type' of a
function or closure. This can lead to unsafe or inefficient behaviors,
where names are captured in a closure that is then communicated, or shared
by multiple threads. Essentially, the problem is that names are *too*
expressive. We can use references in ways their referents cannot be used.
We can put the "gorilla" in the mailbox, but not the gorilla.
This safety issue is especially relevant for RDP. I make heavy use of both
location types ('where' is the value) and substructural types (functions
that cannot be dropped, or cannot be copied, or both). Tacit concatenative
makes safety-by-construction much easier.
Tacit concatenative programming CAN model use of names, i.e. in terms of
lookup in an association list. My proposal will use this technique on
occasion. But there is a very strong, visibly obvious distinction between
the reference and the referent - i.e. the reference is a text value, while
the referent is a gorilla!
*THE USER MODEL*
*
*
The tacit concatenative program can be understood as an unbounded stream of
pure `state -> state` rewrite operations. In addition to these operations,
users have freedom to undo, review, replay, and even rewrite their recent
history of actions. Undo can be accomplished by the normal snapshot-replay
mechanisms.
But we don't model the user as awkwardly 'above' the state, apart from it.
Instead, we model the user within the state. Literally.
(world * user) -> (world * user)
One might think of the 'user' here as the hero of a video-game, and the
'world' as a complex environment that can be navigated or manipulated. The
hero will have hands to carry things, an inventory of loot and weapons,
perhaps a list of special skills. The hero is so important and central to
our model, that navigation is actually modeled by rolling the world under
the hero.
Of course, a user environment isn't a video game. (Or at least it shouldn't
be used that way at all times!) But the same ideas hold.
We may have 'take' and 'put' actions to move objects from the world to the
user. Navigation is often modeled using zipper-like operations through a
document structure, or occasionally by something closer to a hyperlink
(searching for an object by index). Instead of special skills, we have
macros and a powerbox. Instead of loot and weapons, we have projects and
domain-specific toolkits (e.g. paintbrushes, geometry manipulators).
In addition to 'hands', a rather interesting possibility is to have 'eyes'
- programmable lenses that affect how we view, influence, and navigate the
world. Through lenses we might introduce overlays, highlight important
objects, gain x-ray vision for geometries, introduce a head's-up display,
or collapse irrelevant structures.
(NOTE: This hand-and-eye concept - where the hand is programmable by
composable tools, and the eye is programmable by composable lenses, and
this programmability is readily accessible to users - is one I've had in
mind since about 2003.)
*SHARE VALUES NOT ENVIRONMENTS*
A user's environment is extremely personal and personalizable.
Between pattern recognition, code generation, and programming by example, I
imagine that the user and environment will tend to 'grow up' together,
developing a private language specific to each human. In addition, the
environment will acquire a great deal of private information about a user -
e.g. relationships, financial information, pictures and messages.
So users won't want to share their personal environments at that
granularity. And this is fortunate, because they can't. In general, there
is no safe or sensible way to compose independent command streams from
multiple users.
But users can share:
* values - numbers, text, and composites that may represent geometries,
diagrams, documents, graphs, tables, recorded images, sounds, measurements
* behavior-specifying values - e.g. representing macros, lenses, tools, and
authorities
* reactive values - normal or behavioral, time-varying with hidden
dependencies
In RDP systems, sharing between agents occurs via an intermediate resource.
Agents include other humans, but also sensors, databases, and actuators.
The support for reactive time-varying values is a feature provided by RDP,
and involves remaining attached to the value source to track updates.
To share a value, we publish into some space shared with friends or
customers, or a more global space (like a wiki). Private spaces can be
established by a variety of protocols with trusted intermediaries, though
they often must be bootstrapped in physical space.
Not every user thinks about programming, or makes an effort to create
something reusable. But I think most people will fiddle, find interesting
ways to arrange lenses, rearrange documents, smash values together to
create new value. Mashups will be the norm. And even people who aren't
making any effort might be provided useful tools
Everyone is a programmer some of the time.
*ENVIRONMENT METAPHORS FOR USERS?*
I haven't started on the details for a user environment metaphor.
The environments I've developed so far are still aimed at programmers in a
text-based environment. I would probably be focused on a single stack if
RDP didn't have declarative concurrency properties. (A single stack is
painful for modeling concurrent tasks or workflows that must join or synch
at some steps.)
But, based on my interests, I would focus on the following features:
* zoomable user interfaces with live documents
* diagrams, geometries, images, graphs, scene-graphs
* animated non-reactive values (video, GIFs, sound, etc.)
* widgets, variations suitable for use in RDP
* augmented reality systems (visual fingerprints, etc.)
What I can say is this: expressiveness will not be the issue here. We could
model hypermedia systems, desktop metaphor, or whatever else we decide.
The main difference from today's design would be that these are now
constructed of fine-grained values, subject to introspection and
reorganization and mashup, accessible for macro programming, and coexisting
in a common language-based security model.
*ENVIRONMENT IS ALSO A LIVE PROGRAM*
Macros, tools, and so on are designed for volatile manipulation of state.
But that manipulation of state should be meaningful! And to provide meaning
to state, we must use an interpreter. But this interpretation should be
live: as we continue to maintain the state, the meaning should be
propagated automatically.
Here are a few principles that are guiding my thoughts on this subject.
(1) Users must be able to assign their own, private meanings to state in
their personal environments. Each graph, diagram, document, geometry, and
so on can have a different meaning. Some of those meanings will be realized
by programmatic interpretation.
(2) ALL long-running behaviors and policies should have corresponding state
in the environment. Every relationship, shared value, observation on
reactive state, and so on should be accessible in this manner. This is
essential for visibility, maintenance, and for revocation.
(3) Failure is ideally very coarse-grained. Dealing with partial-success is
painful, complicated, and error-prone; we would greatly benefit from
precise atomic success/fail boundaries.
It's addressing these principles where RDP really shines. RDP is based upon
continuous influence and observation, and also has very nice properties for
runtime update and revocation. For clean failure, RDP enables time-warp
style 'undo' even in an open system. Of course, there are practical latency
limits on this (can't always correct the past), but those are partially
addressed: RDP also enables speculative evaluation, so we can tentatively
feel out 'what would happen if'.
So, how do we model this separation?
My current thought is that, since meaning is private to the user, the
association between meanings and objects in the environment should be
maintained as part of the user-model. I'm currently envisioning a very KISS
model: there is an association list at a standard location in the
user-model of a form similar to:
("@foo" * [block interpreting foo])
Then, in the environment, users will have ("@foo" * fooStructure) objects
scattered around with no particular organization. If the whole foo object
is inside some larger structure, like "@bar", then it would be the
prerogative of the bar interpreter to either ask for a foo interpreter or
provide its own interpretation.
In order to enforce the "ALL long running behaviors are modeled in visible
state" principle, the initial program has no authority; it's ultimately
just a sequence of pure state->state transforms. Capabilities are
introduced only the second phase. The real argument to the block
interpreting foo is a pair: (powerblock * fooStructure).
(I'm not entirely satisfied here. In particular, I'd like to have more
precise understanding of source-stable uniqueness for the powerblock.)
Potentially, this entire process might staged, e.g. if the *output* of
interpreting foo contains an ("@baz" * bazStructure)
In this design, text-based programming can still be supported, but
certainly isn't necessary.
*SERIALIZATION: ONE CODE TO BIND THEM*
*
*
I propose that all values be shared by a pure, tacit concatenative
bytecode. There are several reasons for this.
(1) a uniform serialization model will avoid a lot of redundant parser code
and discontinuity spikes. And in practice, a tacit concatenative bytecode
is likely to operate more efficiently than most parsers: it reduces to a
simple series of table lookups (or even a switch expression) and a small
state machine to deal with text and blocks.
(2) in a reactive model like RDP, we often have large structural values
(like an array or scene-graph) where only a few values change. Rather than
sending the whole structure to communicate a small change, this is easily
modeled in terms of streaming more bytecode to operate on the original
value.
(3) we can gain a lot of efficiency by a very simple trick: instead of just
a value, we can operate on a `(value*context)` pair. The context is a
'communication context' that can hold a small library of functions, some
memoized computations, and so on. Functions in the context can be compiled
by the recipient.
(4) code can contain useful assertions, self-validation.
(5) a high level of semantic compression can be achieved without any
additional designs or layers. Though, if semantic compression isn't used,
then regular streaming compression should work pretty well.
I am developing Awelon Bytecode (ABC) for this purpose.
ABC is a typed, tacit, concatenative bytecode for an idealized RDP system.
ABC has very little structure; it is a UTF-8 stream with very few parse
modes:
slsls - (START) normal bytecode mode
{text goes here} - text mode
\}, \\, etc. - escapes in text require a mode
[slsls] - block mode, forming a function
In this case `slsls` means `swap assocl swap assocl swap` - a fairly common
operation that I usually give the name `assocr`.
ABC has a minimal set of primitives and very few types. It's up to a decent
compiler or interpreter to simplify data-plumbing like slsls. The goal with
ABC is not a most efficient direct-interpretation. I am more interested in
keeping things minimal, easy to prove, easy to generate, and easy to
optimize.
ABC has only one syntax-layer value type, which is text. Numbers
(rationals) are specified in ABC by first using text then translating it to
a number. This isn't the ideal representation for efficiency, but I feel
that legibility and simplicity has greater value.
{text} :: x -> (text*x)
# :: (text*x)->num*x
{Hello, World!}
{42}#
Structured values can be formed by constructing elements and organizing
them in a streaming fashion. There are some simple strategies to achieve
this.
(42,108) => {108}#se{42}#
Text can also be used as a comment. I can think of a few reasons this might
be done - e.g. to provide optimizer suggestions, record profiling
information, or potential hints for a theorem prover.
% :: (text*x) -> x
{this is a comment}%
ABC is designed for capability-based languages. I.e. there is no ambient
authority (except for 'error'). Developers can't even create 'unique'
objects (or local state) without a capability.
$ :: (text*x)->cap*x
The interpretation of the text within a serialized capability is entirely
up to the provider of the capability. It could be encrypted code. It could
be HMAC authenticated code. It could be a random GUID to a stored value.
And so on.
In RDP systems, all capabilities are implicitly revocable: to 'grant' a
capability is a continuous action, so to revoke you simply stop granting.
No state is required, and this can be implemented by a variety of
strategies.
ABC doesn't track any pure/impure type. However, RDP has a concept of
location, called 'partition type', which can be used to isolate some
subprograms.
ABC can have spaces, tabs, newlines, and carriage returns. Those all have
the same meaning: identity function.
*WHAT FAILED BEFORE?*
Similar efforts have come and gone, often with some small success that
could not be scaled. My hypothesis is that the following have been points
of failure:
1. Did not model user/programmer. User sits awkwardly above model, no
semantic-layer ability to manipulate it or program-by-example. Wall of
syntax.
2. Second-class extensions. Brushes, tools, lenses, views are not
first-class objects that can be carried and composed. Boiler-plate
namespace management to reuse tools from one task to another.
3. Did not effectively address value sharing, independent maintenance,
security properties. Programmers forced to "ship the IDE" to share behavior.
I believe that all three points must be addressed simultaneously to have
any hope for success. If we address 1,2 we have isolated users - a powerful
environment but no leverage. If we address 3, we have more effective
programmers, but it's all arcane knowledge and hidden APIs.
In my design, points 1,2 are addressed by the tacit concatenative model of
programmer manipulating environment. Point 3 is handled by RDP and
capability security.
profound: a new perspective on an old idea, with consequences deeper and
more pervasive than I had imagined.
The idea is simply this: every user action is an act of meta-programming.
More precisely:
(1) Each user event addends a tacit concatenative program.
(2) The output of the tacit concatenative program is another program.
(3) We can understand the former as rewriting parts of the latter.
(4) These rewrites include the user-model - navigation, clipboard, etc.
I will further explain this idea, why it is powerful, how it is different.
To clarify, this isn't another hand-wavy 'shalt' and 'must' proposal with
no idea of how to achieve it. Hammering at a huge list of requirements for
eight years got me to RDP. At this point, I have concrete ideas on how to
accomplish everything I'm about to describe.
Users Are Programmers.
The TUNES vision is revived, and better than ever.
*WHY TACIT CONCATENATIVE?*
Concatenative programming is perhaps best known through FORTH. Most
concatenative languages have followed in Charles Moore's forthsteps,
sticking with the basic stack concept but focusing on higher-order
programming, types, and other features.
A stack would be an extremely impoverished and cramped environment for a
user; even many programmers would not tolerate it. Fortunately, we can move
beyond the stack environment. And I insist that we do! Concatenative
programming can also be based upon such structures as trees, Huet zippers,
and graphs. This proposal is based primarily on tree-structured data and
zippers, with just a little indirect graph modeling through shared state or
explicit labels (details later).
A 'tacit' programming language is one that does not mention names for
parameters or local variables. Many concatenative programming languages are
also tacit, though the concepts don't fully intersect.
A weakness of tacit concatenative programming is that, in a traditional
text-based programming environment, users must visualize the environment
(stack or other structure) in their head, and that they must memorize a
bunch of arcane 'stack shuffling' words. By comparison, variable names in
text are easy to visualize and review.
My answer: change programming environments!
Powerful advantages of tacit concatenative programming include:
1. the environment has a precisely defined, visualizable value
2. short strings of tacit concatenative code are easy to generate
3. concatenative code is sequential, forming an implicit timeline
4. code also subject to learning, pattern recognition, and rewrites
5. every step, small and large, is precisely defined and typed
Instead of an impoverished, text-based programming environment, we should
offer continuous automatic visualization. Rather than asking users to
memorize arcane words, we should offer direct manipulation: e.g. take, put,
slide, toss, drag, drop, copy, paste. Appropriate tacit concatenative code
is generated at every step, for every gesture. This code is easy to
generate because generators can focus on short vectors of learned 'known
useful' words without syntactic noise; this is subject to a variety of
well-known solutions (logical searches, genetic programming,
hill-climbing).
And then there are benefits that move beyond anything offered today, UI or
PL.
Not only can we visualize the environment, we can animate it. Users can
review and replay their actions, potentially from different perspectives or
highlighting different objects. Since even the smallest dataflow steps are
well defined, users can review at different temporal scales, based on the
granularity of their actions - zooming in to see precisely what taking an
object entails, or zooming out to see broad changes in an environment.
Rewrites can be used to make these animations smoother, more efficient, and
perhaps more aesthetically pleasing. And, in addition to undo, users can
rewrite parts of their history to better understand a decision or to fix a
mistake.
The programming environment can also help users construct macros: pattern
recognition is easy with tacit programming even if it were just in terms of
sequences of words. However, patterns are augmented further by looking at
context, the environment at the time a word was used. Proposed words can be
refined with very simple decision procedures to account for slight
context-sensitive variations. Discovered patterns can be used for simple
compression of history, or be used for programming-by-example.
An environment that recognizes a pattern might quietly and unobtrusively
offer a constructed tool or macro, that the user might refine a little
(e.g. clarifying the decision procedure) before using. The notion of
'dialecting' and 'DSLs' is replaced by problem-specific toolboxes and
macros, where a tool may act a lot like a paintbrush.
Further, there are advantages from the formalism and typing!
For one example, it is to guide user actions relevant to the typeful
context - i.e. making appropriate suggestions. Also, multiple actions can
be assigned to a single gesture or voice command, so long as they are
distinguishable in most typeful contexts. (When there seems to be
ambiguity, the environment can ask for clarification. Not a problem so long
as it's rare.)
By introspecting the environment, we can also create words that are 'smart'
about their application, i.e. automatically performing a search of the
local environment to find an appropriate target, and perhaps validate that
it is a unique target. This ability to be selectively imprecise can greatly
reduce the burden on users and developers. (Usefully, we can separate the
'search' and 'apply' patterns such that augmenting any action with search
is a simple composition.)
Tacit concatenative programming is *safer* than names.
With parameter based programming, the data-plumbing is untyped and ad-hoc.
Further, captured names are almost never visible in the 'type' of a
function or closure. This can lead to unsafe or inefficient behaviors,
where names are captured in a closure that is then communicated, or shared
by multiple threads. Essentially, the problem is that names are *too*
expressive. We can use references in ways their referents cannot be used.
We can put the "gorilla" in the mailbox, but not the gorilla.
This safety issue is especially relevant for RDP. I make heavy use of both
location types ('where' is the value) and substructural types (functions
that cannot be dropped, or cannot be copied, or both). Tacit concatenative
makes safety-by-construction much easier.
Tacit concatenative programming CAN model use of names, i.e. in terms of
lookup in an association list. My proposal will use this technique on
occasion. But there is a very strong, visibly obvious distinction between
the reference and the referent - i.e. the reference is a text value, while
the referent is a gorilla!
*THE USER MODEL*
*
*
The tacit concatenative program can be understood as an unbounded stream of
pure `state -> state` rewrite operations. In addition to these operations,
users have freedom to undo, review, replay, and even rewrite their recent
history of actions. Undo can be accomplished by the normal snapshot-replay
mechanisms.
But we don't model the user as awkwardly 'above' the state, apart from it.
Instead, we model the user within the state. Literally.
(world * user) -> (world * user)
One might think of the 'user' here as the hero of a video-game, and the
'world' as a complex environment that can be navigated or manipulated. The
hero will have hands to carry things, an inventory of loot and weapons,
perhaps a list of special skills. The hero is so important and central to
our model, that navigation is actually modeled by rolling the world under
the hero.
Of course, a user environment isn't a video game. (Or at least it shouldn't
be used that way at all times!) But the same ideas hold.
We may have 'take' and 'put' actions to move objects from the world to the
user. Navigation is often modeled using zipper-like operations through a
document structure, or occasionally by something closer to a hyperlink
(searching for an object by index). Instead of special skills, we have
macros and a powerbox. Instead of loot and weapons, we have projects and
domain-specific toolkits (e.g. paintbrushes, geometry manipulators).
In addition to 'hands', a rather interesting possibility is to have 'eyes'
- programmable lenses that affect how we view, influence, and navigate the
world. Through lenses we might introduce overlays, highlight important
objects, gain x-ray vision for geometries, introduce a head's-up display,
or collapse irrelevant structures.
(NOTE: This hand-and-eye concept - where the hand is programmable by
composable tools, and the eye is programmable by composable lenses, and
this programmability is readily accessible to users - is one I've had in
mind since about 2003.)
*SHARE VALUES NOT ENVIRONMENTS*
A user's environment is extremely personal and personalizable.
Between pattern recognition, code generation, and programming by example, I
imagine that the user and environment will tend to 'grow up' together,
developing a private language specific to each human. In addition, the
environment will acquire a great deal of private information about a user -
e.g. relationships, financial information, pictures and messages.
So users won't want to share their personal environments at that
granularity. And this is fortunate, because they can't. In general, there
is no safe or sensible way to compose independent command streams from
multiple users.
But users can share:
* values - numbers, text, and composites that may represent geometries,
diagrams, documents, graphs, tables, recorded images, sounds, measurements
* behavior-specifying values - e.g. representing macros, lenses, tools, and
authorities
* reactive values - normal or behavioral, time-varying with hidden
dependencies
In RDP systems, sharing between agents occurs via an intermediate resource.
Agents include other humans, but also sensors, databases, and actuators.
The support for reactive time-varying values is a feature provided by RDP,
and involves remaining attached to the value source to track updates.
To share a value, we publish into some space shared with friends or
customers, or a more global space (like a wiki). Private spaces can be
established by a variety of protocols with trusted intermediaries, though
they often must be bootstrapped in physical space.
Not every user thinks about programming, or makes an effort to create
something reusable. But I think most people will fiddle, find interesting
ways to arrange lenses, rearrange documents, smash values together to
create new value. Mashups will be the norm. And even people who aren't
making any effort might be provided useful tools
Everyone is a programmer some of the time.
*ENVIRONMENT METAPHORS FOR USERS?*
I haven't started on the details for a user environment metaphor.
The environments I've developed so far are still aimed at programmers in a
text-based environment. I would probably be focused on a single stack if
RDP didn't have declarative concurrency properties. (A single stack is
painful for modeling concurrent tasks or workflows that must join or synch
at some steps.)
But, based on my interests, I would focus on the following features:
* zoomable user interfaces with live documents
* diagrams, geometries, images, graphs, scene-graphs
* animated non-reactive values (video, GIFs, sound, etc.)
* widgets, variations suitable for use in RDP
* augmented reality systems (visual fingerprints, etc.)
What I can say is this: expressiveness will not be the issue here. We could
model hypermedia systems, desktop metaphor, or whatever else we decide.
The main difference from today's design would be that these are now
constructed of fine-grained values, subject to introspection and
reorganization and mashup, accessible for macro programming, and coexisting
in a common language-based security model.
*ENVIRONMENT IS ALSO A LIVE PROGRAM*
Macros, tools, and so on are designed for volatile manipulation of state.
But that manipulation of state should be meaningful! And to provide meaning
to state, we must use an interpreter. But this interpretation should be
live: as we continue to maintain the state, the meaning should be
propagated automatically.
Here are a few principles that are guiding my thoughts on this subject.
(1) Users must be able to assign their own, private meanings to state in
their personal environments. Each graph, diagram, document, geometry, and
so on can have a different meaning. Some of those meanings will be realized
by programmatic interpretation.
(2) ALL long-running behaviors and policies should have corresponding state
in the environment. Every relationship, shared value, observation on
reactive state, and so on should be accessible in this manner. This is
essential for visibility, maintenance, and for revocation.
(3) Failure is ideally very coarse-grained. Dealing with partial-success is
painful, complicated, and error-prone; we would greatly benefit from
precise atomic success/fail boundaries.
It's addressing these principles where RDP really shines. RDP is based upon
continuous influence and observation, and also has very nice properties for
runtime update and revocation. For clean failure, RDP enables time-warp
style 'undo' even in an open system. Of course, there are practical latency
limits on this (can't always correct the past), but those are partially
addressed: RDP also enables speculative evaluation, so we can tentatively
feel out 'what would happen if'.
So, how do we model this separation?
My current thought is that, since meaning is private to the user, the
association between meanings and objects in the environment should be
maintained as part of the user-model. I'm currently envisioning a very KISS
model: there is an association list at a standard location in the
user-model of a form similar to:
("@foo" * [block interpreting foo])
Then, in the environment, users will have ("@foo" * fooStructure) objects
scattered around with no particular organization. If the whole foo object
is inside some larger structure, like "@bar", then it would be the
prerogative of the bar interpreter to either ask for a foo interpreter or
provide its own interpretation.
In order to enforce the "ALL long running behaviors are modeled in visible
state" principle, the initial program has no authority; it's ultimately
just a sequence of pure state->state transforms. Capabilities are
introduced only the second phase. The real argument to the block
interpreting foo is a pair: (powerblock * fooStructure).
(I'm not entirely satisfied here. In particular, I'd like to have more
precise understanding of source-stable uniqueness for the powerblock.)
Potentially, this entire process might staged, e.g. if the *output* of
interpreting foo contains an ("@baz" * bazStructure)
In this design, text-based programming can still be supported, but
certainly isn't necessary.
*SERIALIZATION: ONE CODE TO BIND THEM*
*
*
I propose that all values be shared by a pure, tacit concatenative
bytecode. There are several reasons for this.
(1) a uniform serialization model will avoid a lot of redundant parser code
and discontinuity spikes. And in practice, a tacit concatenative bytecode
is likely to operate more efficiently than most parsers: it reduces to a
simple series of table lookups (or even a switch expression) and a small
state machine to deal with text and blocks.
(2) in a reactive model like RDP, we often have large structural values
(like an array or scene-graph) where only a few values change. Rather than
sending the whole structure to communicate a small change, this is easily
modeled in terms of streaming more bytecode to operate on the original
value.
(3) we can gain a lot of efficiency by a very simple trick: instead of just
a value, we can operate on a `(value*context)` pair. The context is a
'communication context' that can hold a small library of functions, some
memoized computations, and so on. Functions in the context can be compiled
by the recipient.
(4) code can contain useful assertions, self-validation.
(5) a high level of semantic compression can be achieved without any
additional designs or layers. Though, if semantic compression isn't used,
then regular streaming compression should work pretty well.
I am developing Awelon Bytecode (ABC) for this purpose.
ABC is a typed, tacit, concatenative bytecode for an idealized RDP system.
ABC has very little structure; it is a UTF-8 stream with very few parse
modes:
slsls - (START) normal bytecode mode
{text goes here} - text mode
\}, \\, etc. - escapes in text require a mode
[slsls] - block mode, forming a function
In this case `slsls` means `swap assocl swap assocl swap` - a fairly common
operation that I usually give the name `assocr`.
ABC has a minimal set of primitives and very few types. It's up to a decent
compiler or interpreter to simplify data-plumbing like slsls. The goal with
ABC is not a most efficient direct-interpretation. I am more interested in
keeping things minimal, easy to prove, easy to generate, and easy to
optimize.
ABC has only one syntax-layer value type, which is text. Numbers
(rationals) are specified in ABC by first using text then translating it to
a number. This isn't the ideal representation for efficiency, but I feel
that legibility and simplicity has greater value.
{text} :: x -> (text*x)
# :: (text*x)->num*x
{Hello, World!}
{42}#
Structured values can be formed by constructing elements and organizing
them in a streaming fashion. There are some simple strategies to achieve
this.
(42,108) => {108}#se{42}#
Text can also be used as a comment. I can think of a few reasons this might
be done - e.g. to provide optimizer suggestions, record profiling
information, or potential hints for a theorem prover.
% :: (text*x) -> x
{this is a comment}%
ABC is designed for capability-based languages. I.e. there is no ambient
authority (except for 'error'). Developers can't even create 'unique'
objects (or local state) without a capability.
$ :: (text*x)->cap*x
The interpretation of the text within a serialized capability is entirely
up to the provider of the capability. It could be encrypted code. It could
be HMAC authenticated code. It could be a random GUID to a stored value.
And so on.
In RDP systems, all capabilities are implicitly revocable: to 'grant' a
capability is a continuous action, so to revoke you simply stop granting.
No state is required, and this can be implemented by a variety of
strategies.
ABC doesn't track any pure/impure type. However, RDP has a concept of
location, called 'partition type', which can be used to isolate some
subprograms.
ABC can have spaces, tabs, newlines, and carriage returns. Those all have
the same meaning: identity function.
*WHAT FAILED BEFORE?*
Similar efforts have come and gone, often with some small success that
could not be scaled. My hypothesis is that the following have been points
of failure:
1. Did not model user/programmer. User sits awkwardly above model, no
semantic-layer ability to manipulate it or program-by-example. Wall of
syntax.
2. Second-class extensions. Brushes, tools, lenses, views are not
first-class objects that can be carried and composed. Boiler-plate
namespace management to reuse tools from one task to another.
3. Did not effectively address value sharing, independent maintenance,
security properties. Programmers forced to "ship the IDE" to share behavior.
I believe that all three points must be addressed simultaneously to have
any hope for success. If we address 1,2 we have isolated users - a powerful
environment but no leverage. If we address 3, we have more effective
programmers, but it's all arcane knowledge and hidden APIs.
In my design, points 1,2 are addressed by the tacit concatenative model of
programmer manipulating environment. Point 3 is handled by RDP and
capability security.