[Geany-Devel] [FT-plugins] Proposed "Features"

[Matthew Brush]

Hi All,

Related to my previous mail, and assuming it is somewhat reasonable, I
would like to propose a list of initial features that will be useful to
allow plugins to provide, in no particular order:

----

Syntax Highlighting
-------------------

Most likely using an API based on/similar to Scintilla's "container lexers".

At the minimum, it could have a callback something like:

gboolean (*highlight)(GeanyPlugin*, GeanyDocument*,
guint start_pos, guint end_pos, gpointer user_data);

As with Scintilla's "container lexer", it would just tell the provider
what and where to highlight. It might be pointless providing `end_pos`
it could probably just highlight a whole line at time (maybe like
Scintilla's 'style-needed' notification).

If the providers are to setup their own colour schemes and such, then it
may be required to add a callback such as:

gboolean (*init_styles)(GeanyPlugin*, GeanyDocument*,
gpointer user_data);

for the plugin to configure all the Scintilla styles as it wishes. This
will probably require some kind of conflict avoidance scheme if it is to
support multiple providers providing the same feature. Perhaps it could
also pass the start style ID the provider can use, and it could tell
Geany the maximum style that it ended up using.

Sidebar Symbol Tree
-------------------

Could provide some API functions to populate the tree by plugins
providing the needed information. The nesting could be handled by
passing a scope path similar to GtkTreePath or TagManager's `::`
delimited scope string, which would be parsed/expanded to apply to the
right tree node.

The callback function for the provider might be like:

gboolean (*populate_symbols_tree)(GeanyPlugin*, GeanyDocument*,
guint cursor_pos, gchar **current_node_path /* out */,
gpointer user_data);

When the providers are called in to, they could use a function something
like this:

void tagbar_add_node(const gchar *name, guint line,
const gchar *signature, const gchar *scope_path,
GeanySidebarIcon icon);

I haven't looked closely at the existing sidebar/symbols code yet, maybe
it already provides such a function that could be used. It has also been
mentioned that this could be done using the TagManager API. Plugins
would walk their own internal ASTs or tag lists and build up the TM tag
array, and Geany would use it for updating the symbols tree as it
currently does (IIUC). I don't know much about TM so I can't really give
an example callback for such an API.

Auto-Completion/Intellisense
----------------------------

This could be triggered at the exact same time in the same way it is now
(and re-using the same preferences). It could call a callback function
in the ft-plugins like:

gboolean (*complete_at)(GeanyPlugin *, GeanyDocument *,
guint position, const gchar *partial_word,
GPtrArray completion_list /* out */, gpointer user_data);

The `completion_list` would be filled in with whatever the provider
thinks could be completed at `position`. It could be made more advanced
in the future like allowing plugins to give icons, argument lists,
documentation comment text, etc. For now it could just give strings to
directly populate Scintilla's auto-complete listbox.

Calltips
--------

To provide calltips, a provider could be called into like:

gboolean (*provide_calltips)(GeanyPlugin*, GeanyDocument*,
guint position, const gchar *symbol,
GPtrArray *overloads /* out */, gpointer user_data);

The `overloads` array would populated by the provider with the various
overloads for the given `symbol` (function). As with auto-completion, at
first this could just be a list of overloads represented as strings
which directly populate Scintilla's calltip infos. This could be
enhanced in the future to provide documentation comment text and
placeholder insertion points into the editor in the future, but for now
should be kept simple (see GtkSourceView, XCode, etc).

Go To Declaration/Definition
----------------------------

This could be either single callback passing a parameter to tell whether
declaration or definition, or two separate callbacks. An example of the
callbacks could be:

struct SourceLocation { gchar *filename; guint position; };

gboolean (*get_declaration_location)(GeanyPlugin*, GeanyDocument*,
guint position, const gchar *symbol,
GPtrArray *results, gpointer user_data);

and likewise for `get_definition_location`. The `results` array could be
populated with `SourceLocation`s for each possible
definition/declaration. For example there might be multiple declarations
in different `#if` blocks you could jump to, for C-like languages. Geany
already provides a UI popup list to pick where to jump, this should be
re-used.

Build/Run Support
-----------------

I haven't thought much on this yet, but in general there should be a way
for ft-plugins to get called into when a the build/run features are
activated.

Diagnostics
-----------

I also haven't thought much on this yet. It needs to be possible for
ft-plugins, after getting notified of build/run events, to be able to
update diagnostics information.

Some of the diagnostics display might include:

- Scintilla marker margin icons, indicating the type of diagnostic info
present for each line (could be more than one per line).
- Scintilla indicators (little coloured squigglies) showing the specific
range(s) of code where the diagnostics apply.
- Scintilla annotations giving further details on request.
- Tooltip messages when hovering over the marker margins or squigglied
text giving further details.
- Putting lines of diagnostics info in the msgwin compiler tab (already
supported to some extent IIUC).

Refactoring
-----------

This is just an idea that is probably long in the future, but ft-plugins
could provide some common re-factorings (ex. rename, generate definition
code, etc). I haven't thought on this much at all yet.

----

Almost surely I have left some out. What do you think?

Cheers,
Matthew Brush

[Jiří Techet]

Post by Matthew Brush
Hi All,
Related to my previous mail, and assuming it is somewhat reasonable, I
would like to propose a list of initial features that will be useful to
----
Syntax Highlighting
-------------------
Most likely using an API based on/similar to Scintilla's "container lexers".
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*,
guint start_pos, guint end_pos, gpointer user_data);
As with Scintilla's "container lexer", it would just tell the provider
what and where to highlight. It might be pointless providing `end_pos` it
could probably just highlight a whole line at time (maybe like Scintilla's
'style-needed' notification).
If the providers are to setup their own colour schemes and such, then it
gboolean (*init_styles)(GeanyPlugin*, GeanyDocument*,
gpointer user_data);
for the plugin to configure all the Scintilla styles as it wishes. This
will probably require some kind of conflict avoidance scheme if it is to
support multiple providers providing the same feature. Perhaps it could
also pass the start style ID the provider can use, and it could tell Geany
the maximum style that it ended up using.
Sidebar Symbol Tree
-------------------
Could provide some API functions to populate the tree by plugins providing
the needed information. The nesting could be handled by passing a scope
path similar to GtkTreePath or TagManager's `::` delimited scope string,
which would be parsed/expanded to apply to the right tree node.
gboolean (*populate_symbols_tree)(GeanyPlugin*, GeanyDocument*,
guint cursor_pos, gchar **current_node_path /* out */,
gpointer user_data);
When the providers are called in to, they could use a function something
void tagbar_add_node(const gchar *name, guint line,
const gchar *signature, const gchar *scope_path,
GeanySidebarIcon icon);
I haven't looked closely at the existing sidebar/symbols code yet, maybe
it already provides such a function that could be used. It has also been
mentioned that this could be done using the TagManager API. Plugins would
walk their own internal ASTs or tag lists and build up the TM tag array,
and Geany would use it for updating the symbols tree as it currently does
(IIUC). I don't know much about TM so I can't really give an example
callback for such an API.
Auto-Completion/Intellisense
----------------------------
This could be triggered at the exact same time in the same way it is now
(and re-using the same preferences). It could call a callback function in
gboolean (*complete_at)(GeanyPlugin *, GeanyDocument *,
guint position, const gchar *partial_word,
GPtrArray completion_list /* out */, gpointer user_data);
The `completion_list` would be filled in with whatever the provider thinks
could be completed at `position`. It could be made more advanced in the
future like allowing plugins to give icons, argument lists, documentation
comment text, etc. For now it could just give strings to directly populate
Scintilla's auto-complete listbox.
Calltips
--------
gboolean (*provide_calltips)(GeanyPlugin*, GeanyDocument*,
guint position, const gchar *symbol,
GPtrArray *overloads /* out */, gpointer user_data);
The `overloads` array would populated by the provider with the various
overloads for the given `symbol` (function). As with auto-completion, at
first this could just be a list of overloads represented as strings which
directly populate Scintilla's calltip infos. This could be enhanced in the
future to provide documentation comment text and placeholder insertion
points into the editor in the future, but for now should be kept simple
(see GtkSourceView, XCode, etc).
Go To Declaration/Definition
----------------------------
This could be either single callback passing a parameter to tell whether
declaration or definition, or two separate callbacks. An example of the
struct SourceLocation { gchar *filename; guint position; };
gboolean (*get_declaration_location)(GeanyPlugin*, GeanyDocument*,
guint position, const gchar *symbol,
GPtrArray *results, gpointer user_data);
and likewise for `get_definition_location`. The `results` array could be
populated with `SourceLocation`s for each possible definition/declaration.
For example there might be multiple declarations in different `#if` blocks
you could jump to, for C-like languages. Geany already provides a UI popup
list to pick where to jump, this should be re-used.
Build/Run Support
-----------------
I haven't thought much on this yet, but in general there should be a way
for ft-plugins to get called into when a the build/run features are
activated.
Diagnostics
-----------
I also haven't thought much on this yet. It needs to be possible for
ft-plugins, after getting notified of build/run events, to be able to
update diagnostics information.
- Scintilla marker margin icons, indicating the type of diagnostic info
present for each line (could be more than one per line).
- Scintilla indicators (little coloured squigglies) showing the specific
range(s) of code where the diagnostics apply.
- Scintilla annotations giving further details on request.
- Tooltip messages when hovering over the marker margins or squigglied
text giving further details.
- Putting lines of diagnostics info in the msgwin compiler tab (already
supported to some extent IIUC).
Refactoring
-----------
This is just an idea that is probably long in the future, but ft-plugins
could provide some common re-factorings (ex. rename, generate definition
code, etc). I haven't thought on this much at all yet.
----
Almost surely I have left some out. What do you think?
Cheers,
Matthew Brush
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

Hi Matthew,

sounds good. This is much more lightweight than how #1195 and various other
discussions sounded, I'm happy :-).

All the described functions look good to me in principle except the sidebar
symbol tree one. The current code is quite complex (I was fixing some minor
issues there and spent a lot of time thinking if it could be simplified but
I don't think so). The problem is this: if a file is reparsed and you get
new symbols, if you just cleared the tree and repopulated it with the new
tags, the tree would jump to the beginning no matter where the scrollbar
was before. So what the code does now is it compares the old and new tags,
removes the tags that aren't present in the new tags array and inserts new
tags which weren't present in the old tag array to the right position. If
you have a look at the code, this isn't probably something you want to
write in a plugin :-). So a plugin should provide the full list of new tags
so the tree can do all this diffing and updating.

Which brings me to a question - do you plan to generate TMTag(s) and feed
them to the tag manager instead of the ctags ones? It shouldn't be that
hard and if you do this, you could have the sidebar symbols updated for
free.

These fields in TMTag are unused, don't waste time filling them:

gboolean local; /**< Is the tag of local scope */
guint pointerOrder;
char *inheritance; /**< Parent classes */
char access; /**< Access type (public/protected/private/etc.) */
char impl; /**< Implementation (e.g. virtual) */

Cheers,

Jiri

[Colomban Wendling]

Post by Jiří Techet
[…]
sounds good. This is much more lightweight than how #1195 and various
other discussions sounded, I'm happy :-).

Agreed :) I was a little afraid of seeing a proposal introducing a
gazillion GObject interfaces and GIO extension points ^^

Post by Jiří Techet
All the described functions look good to me in principle except the
sidebar symbol tree one. The current code is quite complex (I was fixing
some minor issues there and spent a lot of time thinking if it could be
simplified but I don't think so). The problem is this: if a file is
reparsed and you get new symbols, if you just cleared the tree and
repopulated it with the new tags, the tree would jump to the beginning
no matter where the scrollbar was before. So what the code does now is
it compares the old and new tags, removes the tags that aren't present
in the new tags array and inserts new tags which weren't present in the
old tag array to the right position. If you have a look at the code,
this isn't probably something you want to write in a plugin :-). So a
plugin should provide the full list of new tags so the tree can do all
this diffing and updating.

Agreed. And the tree updating avoids scroll movements, flickering and
also proven significantly faster than re-building the whole tree (well,
than the previous code at least).

Post by Jiří Techet
Which brings me to a question - do you plan to generate TMTag(s) and
feed them to the tag manager instead of the ctags ones? It shouldn't be
that hard and if you do this, you could have the sidebar symbols updated
for free.

I don't know if plugins should fill the TagManager with extra tags, but
I agree that plugins should probably use the TMTag structure to pass
Tag-like data to Geany. For example, for symbols tree, autocompletion
suggestions and calltips: instead of providing a list of strings,
provide a list of TMTags. As I see it, a TMTag structure contains
everything useful for the current feature set (and could be extended),
and are a fairly canonical representation of it.
A plugin could create temporary TMTags just to give to Geany, or
maintain its own list (or feed TagManager, if we wanted), but anyway
just pass the ones it want to the various APIs.

TMTags contain name, scope (useful for symbol tree and currently
calltips), type (useful for icons at least), signature, etc. All we
need in various places. And a plugin could leave many fields empty if
it doesn't care about them, not many fields are actually required.

Regards,
Colomban

[Thomas Martitz]

Post by Colomban Wendling

Post by Jiří Techet
Which brings me to a question - do you plan to generate TMTag(s) and
feed them to the tag manager instead of the ctags ones? It shouldn't be
that hard and if you do this, you could have the sidebar symbols updated
for free.

I don't know if plugins should fill the TagManager with extra tags, but
I agree that plugins should probably use the TMTag structure to pass
Tag-like data to Geany. For example, for symbols tree, autocompletion
suggestions and calltips: instead of providing a list of strings,
provide a list of TMTags. As I see it, a TMTag structure contains
everything useful for the current feature set (and could be extended),
and are a fairly canonical representation of it.
A plugin could create temporary TMTags just to give to Geany, or
maintain its own list (or feed TagManager, if we wanted), but anyway
just pass the ones it want to the various APIs.
TMTags contain name, scope (useful for symbol tree and currently
calltips), type (useful for icons at least), signature, etc. All we
need in various places. And a plugin could leave many fields empty if
it doesn't care about them, not many fields are actually required.

I think we're on the same page here. I'm also suggesting that we build a
framework which passes tags (or tag-like as you name it) from plugins to
Geany, and Geany uses that to implement features X and Y. If the TMTag
structure is currently insufficient then, well, just extend it as needed.

This is flexible and elegant, and allows to develop new feature (inside
Geany or non-ft-plugins) without changing all the ft-plugins as well.

Best regards.

[Matthew Brush]

Post by Colomban Wendling

Post by Jiří Techet
[…]
Which brings me to a question - do you plan to generate TMTag(s) and
feed them to the tag manager instead of the ctags ones? It shouldn't be
that hard and if you do this, you could have the sidebar symbols updated
for free.

I don't know if plugins should fill the TagManager with extra tags, but
I agree that plugins should probably use the TMTag structure to pass
Tag-like data to Geany. For example, for symbols tree, autocompletion
suggestions and calltips: instead of providing a list of strings,
provide a list of TMTags. As I see it, a TMTag structure contains
everything useful for the current feature set (and could be extended),
and are a fairly canonical representation of it.
A plugin could create temporary TMTags just to give to Geany, or
maintain its own list (or feed TagManager, if we wanted), but anyway
just pass the ones it want to the various APIs.
TMTags contain name, scope (useful for symbol tree and currently
calltips), type (useful for icons at least), signature, etc. All we
need in various places. And a plugin could leave many fields empty if
it doesn't care about them, not many fields are actually required.

I feel like it would be easier to provide a small API to fill the tree
(and even provide this - surprisingly to me - complicated code to make
the treeview do what it's main purpose is), and that filling in TM tags
might be tedious API-wise, but I admittedly don't know enough about TM
and the sidebar code to make this judgment with full confidence.

It seems like you, Jiří and Thomas all agree (to varying degrees) that
it makes sense to re-use TM for some stuff like this, so it's fine with
me. When I update the Github tracker issue with the results of these
discussions, I will include this usage of TM tags (at least).

Cheers,
Matthew Brush

[Thomas Martitz]

Post by Matthew Brush
Hi All,
Related to my previous mail, and assuming it is somewhat reasonable, I
would like to propose a list of initial features that will be useful

I disagree with this approach. This adds per use case hooks to plugins,
which then became part of the stable API. I don't think that we have to
codify every single use case of tags into the plugins. That's just
making it harder (maybe impossible?) to change or add use cases.

I thought we agreed that plugin should simply provide tags to Geany/TM
and then Geany uses them for various purposes as it sees fit (possibly
depending on user preferences), even if it means to extend TM in a few
places.

Sorry, I will not support the approach you are suggesting, until it's
clear to me that it's not doable otherwise. Instead, we should inspect
each use case and locate exact requirements for "tag passing".

PS: highlighting and tags are generally separated topics. With custom
filetypes we can have highlighting even if we don't have a parser. I'd
rather not mix highlighting in this TM-related topics. Though I could
imagine we could make it work with some magic tags that provide keywords
to be highlighted, or just simply make use of dynamic lexers.

Best regards.

[Lex Trotman]

This adds per use case hooks to plugins, which then became part of the stable API. I don't think that we have to codify every single use case of tags into the plugins. That's just making it harder (maybe impossible?) to change or add use cases.

The point of this proposal is to change and add use-cases that are not
currently possible with the current plugin API. But instead of each
use-case generating its own piece of API and its own infrastructure,
the point of the FT-plugins proposal is to provide a common
infrastructure and approach for all filetype specific use-cases, those
needed for currently suggested uses, indentation, clang based styling
and symbols, and as framework for future use-cases we either havn't
thought of, or havn't a concrete intention to add immediately.

I thought we agreed that plugin should simply provide tags to Geany/TM

This proposal is about many types of filetype specific functionality,
not just tags. Tagmanager will not help in any way with indenting
Haskell, or even C++.

I'd rather not mix highlighting in this TM-related topics.

It appears you have misunderstood the purpose of the proposal, or I have.

It is my understanding that, as noted above, the proposal is to allow
plugins to provide filetype specific capabilities for many areas,
"features" in Matthews parlance. Its not intending to change the
existing plugin API but to extend it to support functionality that
cannot be performed at the moment.

An example is Matthews attempt at using libclang in a plugin, which
provides, highlighting, symbol lookups, formatting etc with much
greater accuracy than the existing facilities in Geany can do. But at
the moment there is no facility in Geany for any of that that
information to be injected for any filetype, or without forcing it to
be limited by Geany built-in capabilities.

Jiri has for example noted he doesn't want to be built into Geany the
type of project management capability needed to allow accurate enough
includes to be determined to allow accurate symbols. Thats fine, but
at the moment its also not possible for anyone to add it in a plugin
either. If Geany is to remain closed to such plugins it will remain
"all languages are poorly handled C" and slip further behind other
tools for newer languages that become more and more "not C". And I
for one would be sad to see that happen simply because of a refusal to
make Geany open enough.

Cheers
Lex

[Thomas Martitz]

This adds per use case hooks to plugins, which then became part of the stable API. I don't think that we have to codify every single use case of tags into the plugins. That's just making it harder (maybe impossible?) to change or add use cases.
The point of this proposal is to change and add use-cases that are not
currently possible with the current plugin API. But instead of each
use-case generating its own piece of API and its own infrastructure,
the point of the FT-plugins proposal is to provide a common
infrastructure and approach for all filetype specific use-cases, those
needed for currently suggested uses, indentation, clang based styling
and symbols, and as framework for future use-cases we either havn't
thought of, or havn't a concrete intention to add immediately.

Post by Thomas Martitz
I thought we agreed that plugin should simply provide tags to Geany/TM

This proposal is about many types of filetype specific functionality,
not just tags. Tagmanager will not help in any way with indenting
Haskell, or even C++.

4 of 5 of the proposed features are strictly tag-related. And Geany can
do all of them already, it's just that the current implementation leaves
things to be desired so there is the idea to let plugins improve upon them.

I disagree with the proposed solution for those 4, because they are
offloading logic on a per feature basis to plugins, only because Geany
isn't capable at the moment. If Geany was capable, then there could be 1
solution for the 4 features and less complexity in each plugin (and we
know the quality of plugins varies a lot so they should have little
complexity as possible).

The solution I have in mind simply allows plugins to pass tags to Geany
which they parsed with more advanced code. The tags itself would
advanced too, to allow for the improvements current TM+ctags can't
offer. Symbol tree, calltips, autocompletion, jump-to-decl can all be
improved based on the advanced tags.

Best regards.

[Matthew Brush]

Post by Thomas Martitz

This adds per use case hooks to plugins, which then became part of the
stable API. I don't think that we have to codify every single use case
of tags into the plugins. That's just making it harder (maybe
impossible?) to change or add use cases.
The point of this proposal is to change and add use-cases that are not
currently possible with the current plugin API. But instead of each
use-case generating its own piece of API and its own infrastructure,
the point of the FT-plugins proposal is to provide a common
infrastructure and approach for all filetype specific use-cases, those
needed for currently suggested uses, indentation, clang based styling
and symbols, and as framework for future use-cases we either havn't
thought of, or havn't a concrete intention to add immediately.

Post by Thomas Martitz
I thought we agreed that plugin should simply provide tags to Geany/TM

This proposal is about many types of filetype specific functionality,
not just tags. Tagmanager will not help in any way with indenting
Haskell, or even C++.

4 of 5 of the proposed features are strictly tag-related. And Geany can
do all of them already, it's just that the current implementation leaves
things to be desired so there is the idea to let plugins improve upon them.
I disagree with the proposed solution for those 4, because they are
offloading logic on a per feature basis to plugins, only because Geany
isn't capable at the moment. If Geany was capable, then there could be 1
solution for the 4 features and less complexity in each plugin (and we
know the quality of plugins varies a lot so they should have little
complexity as possible).

Using TagManager would likely add more complexity to the plugins. Using
the libclang example, it already has its own internal representation of
"tags" (ie. the AST). Making plugins munge their internal representation
to convert/conform to TagManager's would add extra burden.

For example "go to definition/declaration" is a simple task with
libclang, you would just get the source location and pass back to Geany
the filename and position and it could go there easily, without any need
for TagManager as a middle-man. Likewise, libclang has a function to
return valid completions for a given position in the source code,
passing a list of strings for Geany to show in the Scintilla
auto-complete list would be a simple matter of building a string array.
Using tag manager would require building a tags array that probably
wouldn't/shouldn't become part of any larger tags array
(workspace/global/whatever) and then getting Geany to show that tags
array without messing with it (ordering, etc).

Another issue that might arise is that TagManager might very well choke
if it had to deal with every single tag/symbol in the entire source
(make a "hello world" in C++ using iostream and look at the preprocessed
source, for example). The number of tags it would need to store/track
would way more than double when you consider local variables,
parameters, inner-classes, etc. I have no clue if TM is actually able to
handle this at all, efficiently enough. Ontop of that, all this
information is stored twice in memory, once in the libclang AST and once
in TagManager tag arrays. This could result in quite some additional
memory overhead.

Jiří gives an example (the symbols sidebar) where this may be warranted
and ultimately cause less work, since the sidebar updating code is
apparently quite complicated. I still need to study this code to
understand why so much effort is needed to keep the symbol tree at the
same relative scroll location when updated.

Post by Thomas Martitz
The solution I have in mind simply allows plugins to pass tags to Geany
which they parsed with more advanced code. The tags itself would
advanced too, to allow for the improvements current TM+ctags can't
offer. Symbol tree, calltips, autocompletion, jump-to-decl can all be
improved based on the advanced tags.

I'm not strictly opposed to wrangling stuff through TagManager. Could
you provide example hooks showing the kind of interface you would
thinking about for the mentioned features? Also in which cases do you
expect TagManager is not up to the task and would need to be modified?

Cheers,
Matthew Brush

[Jiří Techet]

Post by Thomas Martitz
The solution I have in mind simply allows plugins to pass tags to Geany
which they parsed with more advanced code. The tags itself would advanced
too, to allow for the improvements current TM+ctags can't offer. Symbol
tree, calltips, autocompletion, jump-to-decl can all be improved based on
the advanced tags.

I'm personally very happy that Matthew decided to go the way he proposed
instead of doing what you propose. As I said in one of the posts here

https://github.com/geany/geany/pull/1187

it's not a matter of how "advanced" the tags are but that the tags are
represented in a completely different way - tree vs list. There's info in
the tree that cannot be stored in the list and similarly ctags list cannot
be converted to AST. You can start doing some crazy things like serializing
AST to a list and then deserializing it to a tree or having parallel AST-TM
and list-TM but things are complicated enough in TM already and merging
ASTs during source file updates would be really complicated. Also
performance would suffer because operations on GPtrArrays are really fast
as they don't require pointer dereferencing. In addition I think each
library will have its own partially incompatible AST representation and
different info will be needed e.g. for autocompletion and it will be really
hard to find something generic enough.

So big +1 for Matthew's approach from me. What might be good though is
filling the "simple" TMTags with "advanced" libclang or any other tags into
the existing TM. Some info will be lost on the way but this info will be
sufficient for some things like the symbol tree or your plugin.

Jiri

[Thomas Martitz]

Post by Thomas Martitz
The solution I have in mind simply allows plugins to pass tags to
Geany which they parsed with more advanced code. The tags itself
would advanced too, to allow for the improvements current TM+ctags
can't offer. Symbol tree, calltips, autocompletion, jump-to-decl
can all be improved based on the advanced tags.
I'm personally very happy that Matthew decided to go the way he
proposed instead of doing what you propose. As I said in one of the
posts here
https://github.com/geany/geany/pull/1187
it's not a matter of how "advanced" the tags are but that the tags are
represented in a completely different way - tree vs list. There's info
in the tree that cannot be stored in the list and similarly ctags list
cannot be converted to AST. You can start doing some crazy things like
serializing AST to a list and then deserializing it to a tree or
having parallel AST-TM and list-TM but things are complicated enough
in TM already and merging ASTs during source file updates would be
really complicated. Also performance would suffer because operations
on GPtrArrays are really fast as they don't require pointer
dereferencing. In addition I think each library will have its own
partially incompatible AST representation and different info will be
needed e.g. for autocompletion and it will be really hard to find
something generic enough.

1) trees can be stored in flat list simply by storing the tree path in
each element (like GtkTreeModel). This is not crazy or complex. It's
even readily available in the scope string.
2) I don't see why the AST needs to be recoverable from a TM tag list.
You only need to map from X to TM, so losing information is OK as long
as it's sufficient for the function of current features/use cases (for
future features TM tag list may or may not needed to be enhanced).

Please show me how the proposed tag-related features require advanced
tag management that isn't possible with TM if tags would be supplemented
or completely provided by a plugin.

Post by Thomas Martitz
What might be good though is filling the "simple" TMTags with
"advanced" libclang or any other tags into the existing TM. Some info
will be lost on the way but this info will be sufficient for some
things like the symbol tree or your plugin.

Right, this is basically what I'm looking at. I can't see how we need
anything more at this point. And yes, I do care that Geany itself and
plugins (including mine) have a unified interface to access all tags,
regardless of who provides them. Matthews proposal prohibits this.

There is also another aspect about the proposal that worries me: a
plugin shall provide N features for M languages. And X plugins might be
compete (not even considering the desire that plugins can build upon
each other). This means (slightly exaggerated) that N*M*X possibilities
have to be managed by Geany's conflict resolution. I don't want to
implement that. It seems much simpler to me to collect tags from
plugins, merge them (maybe throw out duplicates) and pass them to the
actual feature code all within Geany.

What worries me is that we jumped from mere brainstorming to a
relatively concrete proposal, without evaluating requirements or any
other research. Or was this evaluation just invisible to me?

Best regards.

[Matthew Brush]

Post by Thomas Martitz

[...]

There is also another aspect about the proposal that worries me: a
plugin shall provide N features for M languages. And X plugins might be
compete (not even considering the desire that plugins can build upon
each other). This means (slightly exaggerated) that N*M*X possibilities
have to be managed by Geany's conflict resolution. I don't want to
implement that. It seems much simpler to me to collect tags from
plugins, merge them (maybe throw out duplicates) and pass them to the
actual feature code all within Geany.

In principle it's not that hard to manage, as mentioned in my "Proposed
Design" message, Geany just needs to keep the providers in a list and
the callbacks work like GTK/GDK callbacks (and some in Geany) where the
callback's return value determines whether the next provider is called
or not. In that message I attached a mockup of a kind of UI that could
be used to allow users absolute control, and for Geany it's just a (set
of) ordered lists.

Post by Thomas Martitz
What worries me is that we jumped from mere brainstorming to a
relatively concrete proposal, without evaluating requirements or any
other research. Or was this evaluation just invisible to me?

I evaluated and experimented with several different approaches and
discussed various details with some of the main developers (including
you) on IRC. Based on the way that in my opinion, as someone who has
tried and failed to implement the needed features in the past, and as a
Geany developer, I recommended a proposed design for further input. And
here we are :)

As I asked in an earlier message, I'd be interested if you could provide
some more concrete examples of what you were thinking with using TM,
which would accomplish the goals mentioned in the Github Issue and
fleshed-out more in the top of this thread.

Cheers,
Matthew Brush

[Thomas Martitz]

Post by Matthew Brush

Post by Thomas Martitz

[...]

There is also another aspect about the proposal that worries me: a
plugin shall provide N features for M languages. And X plugins might be
compete (not even considering the desire that plugins can build upon
each other). This means (slightly exaggerated) that N*M*X possibilities
have to be managed by Geany's conflict resolution. I don't want to
implement that. It seems much simpler to me to collect tags from
plugins, merge them (maybe throw out duplicates) and pass them to the
actual feature code all within Geany.

In principle it's not that hard to manage, as mentioned in my
"Proposed Design" message, Geany just needs to keep the providers in a
list and the callbacks work like GTK/GDK callbacks (and some in Geany)
where the callback's return value determines whether the next provider
is called or not. In that message I attached a mockup of a kind of UI
that could be used to allow users absolute control, and for Geany it's
just a (set of) ordered lists.

You say this should be easy, I say I expect it to be complicated. This
is quite the same (just the other way around) with my suggestion to just
pass tags to Geany. There you keep claiming that it'll be a massive
change why I expect it to be relatively easy to do. At least not harder
than changing 6+ core parts in Geany to execute plugin callbacks and
make something useful from the results.

Post by Matthew Brush

Post by Thomas Martitz
What worries me is that we jumped from mere brainstorming to a
relatively concrete proposal, without evaluating requirements or any
other research. Or was this evaluation just invisible to me?

I evaluated and experimented with several different approaches and
discussed various details with some of the main developers (including
you) on IRC. Based on the way that in my opinion, as someone who has
tried and failed to implement the needed features in the past, and as
a Geany developer, I recommended a proposed design for further input.
And here we are :)

Please show me the point in the IRC logs where I agreed with your
approach. In fact, I can't remember discussing ft-plugins with you on
IRC at all (I just asked at one point how libclang works in more detail).

Your evaluation is still invisible to me.

Post by Matthew Brush
As I asked in an earlier message, I'd be interested if you could
provide some more concrete examples of what you were thinking with
using TM, which would accomplish the goals mentioned in the Github
Issue and fleshed-out more in the top of this thread.

Essentially I'd propose a signal that's emitted when Geany begins
(re-)parsing a file (I believe it does this after each keystroke, after
a debouncing timeout). Plugins could connect to that signal, perhaps
parse the file again, using their special, language specific library
code, and pass the tags they've got to Geany using a to-be-designed API
function (probably involving TMTag and TMSourceFile). Alternatively, the
plugin signal handlers could run before Geany's own one, potentially
blocking Geany from parsing the file at all. Or both approaches.

Geany would then merge the tags, perhaps giving the plugin ones more
weight, and store it in TM.

Then the symbol tree, completion, other plugins can use the tags and
make their stuff.

For highlighting, I would really try to use Scintilla dynamic lexers.
Otherwise we're going to have trouble with inconsistent editor widgets
and non-working color schemes.

Best regards

[Matthew Brush]

Post by Thomas Martitz

Post by Matthew Brush

Post by Thomas Martitz

[...]

There is also another aspect about the proposal that worries me: a
plugin shall provide N features for M languages. And X plugins might be
compete (not even considering the desire that plugins can build upon
each other). This means (slightly exaggerated) that N*M*X possibilities
have to be managed by Geany's conflict resolution. I don't want to
implement that. It seems much simpler to me to collect tags from
plugins, merge them (maybe throw out duplicates) and pass them to the
actual feature code all within Geany.

In principle it's not that hard to manage, as mentioned in my
"Proposed Design" message, Geany just needs to keep the providers in a
list and the callbacks work like GTK/GDK callbacks (and some in Geany)
where the callback's return value determines whether the next provider
is called or not. In that message I attached a mockup of a kind of UI
that could be used to allow users absolute control, and for Geany it's
just a (set of) ordered lists.

You say this should be easy, I say I expect it to be complicated. This
is quite the same (just the other way around) with my suggestion to just
pass tags to Geany. There you keep claiming that it'll be a massive
change why I expect it to be relatively easy to do. At least not harder
than changing 6+ core parts in Geany to execute plugin callbacks and
make something useful from the results.

I've tinkered with implementations, it's generally as easy as walking a
list in a loop, calling a function and breaking early if the function
returns `FALSE`.

I don't think I claimed it will be a massive change, I think I just said
it will add more complexity to ft-plugins, as opposed to less, as you
claimed.

Since the very first response I made to you about TM, I've been open to
the idea. I raised a few concerns which you never responded to, namely
that it may not be possible for TM to handle all the tags in Clang's AST
(it has a very rich and complex AST, heavily optimized by some of the
smartest people in the C++ world). Also the AST is HUGE and representing
it twice in memory would at least (if not more than) double the memory
footprint.

Post by Thomas Martitz

Post by Matthew Brush

Post by Thomas Martitz
What worries me is that we jumped from mere brainstorming to a
relatively concrete proposal, without evaluating requirements or any
other research. Or was this evaluation just invisible to me?

I evaluated and experimented with several different approaches and
discussed various details with some of the main developers (including
you) on IRC. Based on the way that in my opinion, as someone who has
tried and failed to implement the needed features in the past, and as
a Geany developer, I recommended a proposed design for further input.
And here we are :)

Please show me the point in the IRC logs where I agreed with your
approach. In fact, I can't remember discussing ft-plugins with you on
IRC at all (I just asked at one point how libclang works in more detail).

I never said I proposed a design on IRC, that's what the "Proposed
Design" email was about. I discussed approaches and details with some of
the developers. You and I talked about whether TM might be up to the job
of representing scope, for example (and you pointed out that it can,
crudely, by using strings rather than parent/child relationships).

Post by Thomas Martitz
Your evaluation is still invisible to me.

I evaluated the various approaches myself and by discussing specific
topics with some devs lately and previously when we talked about
ft-plugins so that I would be able to propose a design on the mailing
list, and here we are discussing that. I don't see the problem.

Post by Thomas Martitz

Post by Matthew Brush
As I asked in an earlier message, I'd be interested if you could
provide some more concrete examples of what you were thinking with
using TM, which would accomplish the goals mentioned in the Github
Issue and fleshed-out more in the top of this thread.

Essentially I'd propose a signal that's emitted when Geany begins
(re-)parsing a file (I believe it does this after each keystroke, after
a debouncing timeout). Plugins could connect to that signal, perhaps
parse the file again, using their special, language specific library
code, and pass the tags they've got to Geany using a to-be-designed API
function (probably involving TMTag and TMSourceFile). Alternatively, the
plugin signal handlers could run before Geany's own one, potentially
blocking Geany from parsing the file at all. Or both approaches.

Unrelated to whether to use TM or not, I think you're right that a hook
to tell ft-plugins to re-parse would be useful, otherwise plugins will
all have to implement that logic/signal connections themselves.

Post by Thomas Martitz
Geany would then merge the tags, perhaps giving the plugin ones more
weight, and store it in TM.

I think you underestimate how many tags we're talking here. The example
libclang ft-plugin would have to re-walk the entire AST (which is
absolutely massive, particularly for C++), convert it to TM tag
structures, and then Geany/TM would have to perform some merging logic,
would would be more complicated than now if it was to support C++
properly, every single re-parse. My intuition tells me that just won't
be fast enough, Clang already jumps through hoops and uses tricks to
just build its own AST in-time.

Post by Thomas Martitz
Then the symbol tree, completion, other plugins can use the tags and
make their stuff.
For highlighting, I would really try to use Scintilla dynamic lexers.
Otherwise we're going to have trouble with inconsistent editor widgets
and non-working color schemes.

I have tried that, I can try and find the code if I still have it. The
problem I have with it is that it adds a whole other framework (ILexer)
and language (C++) that has to be used, and it's a lot more involved
just to perform basically the same task. In both cases ft-plugins could
either provide their own states or re-use those from an existing lexer
built-into Geany, so I don't think container vs dynamic lexer will have
any bearing on the inconsistency of colour schemes.

Cheers,
Matthew Brush

[Jiří Techet]

Post by Matthew Brush

Post by Thomas Martitz

Post by Matthew Brush

Post by Thomas Martitz

[...]

There is also another aspect about the proposal that worries me: a
plugin shall provide N features for M languages. And X plugins might be
compete (not even considering the desire that plugins can build upon
each other). This means (slightly exaggerated) that N*M*X possibilities
have to be managed by Geany's conflict resolution. I don't want to
implement that. It seems much simpler to me to collect tags from
plugins, merge them (maybe throw out duplicates) and pass them to the
actual feature code all within Geany.

In principle it's not that hard to manage, as mentioned in my
"Proposed Design" message, Geany just needs to keep the providers in a
list and the callbacks work like GTK/GDK callbacks (and some in Geany)
where the callback's return value determines whether the next provider
is called or not. In that message I attached a mockup of a kind of UI
that could be used to allow users absolute control, and for Geany it's
just a (set of) ordered lists.

You say this should be easy, I say I expect it to be complicated. This
is quite the same (just the other way around) with my suggestion to just
pass tags to Geany. There you keep claiming that it'll be a massive
change why I expect it to be relatively easy to do. At least not harder
than changing 6+ core parts in Geany to execute plugin callbacks and
make something useful from the results.

I've tinkered with implementations, it's generally as easy as walking a
list in a loop, calling a function and breaking early if the function
returns `FALSE`.
I don't think I claimed it will be a massive change, I think I just said
it will add more complexity to ft-plugins, as opposed to less, as you
claimed.
Since the very first response I made to you about TM, I've been open to
the idea. I raised a few concerns which you never responded to, namely that
it may not be possible for TM to handle all the tags in Clang's AST (it has
a very rich and complex AST, heavily optimized by some of the smartest
people in the C++ world). Also the AST is HUGE and representing it twice in
memory would at least (if not more than) double the memory footprint.

Post by Thomas Martitz

Post by Matthew Brush
What worries me is that we jumped from mere brainstorming to a

Post by Thomas Martitz
relatively concrete proposal, without evaluating requirements or any
other research. Or was this evaluation just invisible to me?

I evaluated and experimented with several different approaches and
discussed various details with some of the main developers (including
you) on IRC. Based on the way that in my opinion, as someone who has
tried and failed to implement the needed features in the past, and as
a Geany developer, I recommended a proposed design for further input.
And here we are :)

Please show me the point in the IRC logs where I agreed with your
approach. In fact, I can't remember discussing ft-plugins with you on
IRC at all (I just asked at one point how libclang works in more detail).

I never said I proposed a design on IRC, that's what the "Proposed Design"
email was about. I discussed approaches and details with some of the
developers. You and I talked about whether TM might be up to the job of
representing scope, for example (and you pointed out that it can, crudely,
by using strings rather than parent/child relationships).
Your evaluation is still invisible to me.
I evaluated the various approaches myself and by discussing specific
topics with some devs lately and previously when we talked about ft-plugins
so that I would be able to propose a design on the mailing list, and here
we are discussing that. I don't see the problem.

Post by Thomas Martitz

Post by Matthew Brush
As I asked in an earlier message, I'd be interested if you could
provide some more concrete examples of what you were thinking with
using TM, which would accomplish the goals mentioned in the Github
Issue and fleshed-out more in the top of this thread.

Essentially I'd propose a signal that's emitted when Geany begins
(re-)parsing a file (I believe it does this after each keystroke, after
a debouncing timeout). Plugins could connect to that signal, perhaps
parse the file again, using their special, language specific library
code, and pass the tags they've got to Geany using a to-be-designed API
function (probably involving TMTag and TMSourceFile). Alternatively, the
plugin signal handlers could run before Geany's own one, potentially
blocking Geany from parsing the file at all. Or both approaches.

Unrelated to whether to use TM or not, I think you're right that a hook to
tell ft-plugins to re-parse would be useful, otherwise plugins will all
have to implement that logic/signal connections themselves.
Geany would then merge the tags, perhaps giving the plugin ones more

Post by Thomas Martitz
weight, and store it in TM.

I think you underestimate how many tags we're talking here. The example
libclang ft-plugin would have to re-walk the entire AST (which is
absolutely massive, particularly for C++), convert it to TM tag structures,
and then Geany/TM would have to perform some merging logic, would would be
more complicated than now if it was to support C++ properly, every single
re-parse. My intuition tells me that just won't be fast enough, Clang
already jumps through hoops and uses tricks to just build its own AST
in-time.

I think it would be a disaster performance-wise. The number of AST nodes
can be easily 100x more than the amount of tags we have from ctags (we get
a single tag for a function now and AST will contain complete tree for the
function body) so just this might cost 100x more. In addition all the
necessary copies to TM internal representation, having to maintain the tree
structure (in TM we use GPtrArrays for everything which are very efficient
and during tag merge we try to eliminate even pointer dereferences because
those start getting expensive when managing many tags) etc.

And even if we did this, I don't know how we could handle ASTs of different
languages in a generic way because these will differ significantly.

Anyway, if needed we can always add more elements to the TMTag structure so
plugins can add some more information.

Cheers,

Jiri

[Thomas Martitz]

Post by Thomas Martitz
Geany would then merge the tags, perhaps giving the plugin ones more
weight, and store it in TM.
I think you underestimate how many tags we're talking here. The
example libclang ft-plugin would have to re-walk the entire AST
(which is absolutely massive, particularly for C++), convert it to
TM tag structures, and then Geany/TM would have to perform some
merging logic, would would be more complicated than now if it was
to support C++ properly, every single re-parse. My intuition tells
me that just won't be fast enough, Clang already jumps through
hoops and uses tricks to just build its own AST in-time.
I think it would be a disaster performance-wise. The number of AST
nodes can be easily 100x more than the amount of tags we have from
ctags (we get a single tag for a function now and AST will contain
complete tree for the function body) so just this might cost 100x
more. In addition all the necessary copies to TM internal
representation, having to maintain the tree structure (in TM we use
GPtrArrays for everything which are very efficient and during tag
merge we try to eliminate even pointer dereferences because those
start getting expensive when managing many tags) etc.

Let's not outright reject possible solutions based on performance
assumptions and guestimations. Performance can be evaluated on an actual
implementation. Until then it's simply an invalid argument for this
discussion. But FWIW, I don't think performance is the driving aspect.

What's needed from the AST is tags (as you and I mentioned elsewhere,
AST is the complete code representation, so much more than what's
required currently). Those can be extracted in one pass. I don't see
that tags need to be converted back to the original AST.

For any successful plugin operation, the AST has to be generated (and
probably re-generated regularly) and traversed at least once. Creating
tags in a traversal that's happening done anyway probably isn't even
going to add much overhead, if any.

As you say, we use a GPtrArray of tags because it's very efficient for
sorting and merging. I think any ft-plugin will also have to sort (at
least) for showing auto-completion and symbols tree - if it shows them
itself. So it may even have to create such array/lists that TM uses
anyway (you can't sort AST directly). So it might as well pass them to
TM for sorting.

Post by Thomas Martitz
And even if we did this, I don't know how we could handle ASTs of
different languages in a generic way because these will differ
significantly.

One more time, seems I wasn't clear enough yet: I'm *not* suggesting
that we create generic code inside Geany that handles any kind of AST.
What I suggest is that plugins that use AST internally (or not) pass
tag-like information to Geany, extracted and flattened from its internal
AST (or whatever it uses). Then Geany and other plugins can use that
information for their own purposes.

Post by Thomas Martitz
Anyway, if needed we can always add more elements to the TMTag
structure so plugins can add some more information.

Yes. I do want to be able to implement new tag-related features in Geany
or other plugins. So if an ft-plugin improves tags over Geany's builtin
abilities, then these should become generally available.

Otherwise we get into a situation where core features of Geany become
exclusive to individual plugins and cannot be improved upon again (get
stuck with a unmaintained plugin that isn't updated to newer Geany
features, anyone?).

Considering the poor quality and orphaned status of some G-P plugins, I
wouldn't want to give individual plugins exclusive access to otherwise
appreciated features which are mostly working in Geany core as of today.
I'd rather not use ft-plugins at all if they provoke such a situation.

Best regards.

[Matthew Brush]

Post by Thomas Martitz
[...]

Post by Jiří Techet
And even if we did this, I don't know how we could handle ASTs of
different languages in a generic way because these will differ
significantly.

One more time, seems I wasn't clear enough yet: I'm *not* suggesting
that we create generic code inside Geany that handles any kind of AST.
What I suggest is that plugins that use AST internally (or not) pass
tag-like information to Geany, extracted and flattened from its internal
AST (or whatever it uses). Then Geany and other plugins can use that
information for their own purposes.

We do need more than just "tags-like information" though, for example,
what's Geany gonna give for completions here (C++):

auto bar = foo<X>()->...

Or even on the next line:

bar.blah = baz... + 3

In the first case, it needs to know the type of not only `foo`'s return
value, but it needs to know the type of `foo<X>`'s return type (could
depend on `X`'s type and/or lots of other stuff).

For the second line, a super-smart auto-complete might only give
suggestions for a `baz` that has a type that has an `::operator+` member
that works with ints (or `baz` is a primitive numeric type), what's
more, it could also filter the suggestions down so that it only shows
identifiers (in any accessible scope) which start with `baz` and when
the global or member `::operator+` is applied to it and 3, and the
result of that which is also compatible with the `blah` member of `bar`
(which is the type of the return of `foo<x>()` and never even mentioned
locally).

As a further example, if we ever upgrade Geany's auto-complete list to
be more than Scintilla's default (to something similar the one in
GtkSourceView IIRC), which say for example showed the doc-comments for a
given symbol (something libclang provides in/along with the AST, and is
also accessible in Python AST as well, IIRC), not only now does
TagManager have to store all that comment text (redundantly, or else
call into the ft-plugin to get it on demand), but it has to pick-out the
_exact_ right completion for the comment to be correct, based on all
kinds of factors, such as the language's scope lookup rules (even inside
of `if` blocks, which it also now has to store in the Tags array or
recover from the source code some how), but also based on templated
types once interpreted (TM would need to store some kind of decypherable
mangled names after templates are expanded), and overloads (which it
would need to determine given the given set of overloads, based on
argument types which compile (and also SFINAE and all that jazz), and
inheritance rules (TM would need to keep track of polymorphic,
potentially multiply-inherited relationships), etc.

Basically, TM would have to have a semantic analyzer with the strength
of that of a real compiler, for each language an ft-plugin might
implement. That is what's needed for real Intellisense-like features and
I believe what some of the responders are getting at. The same kind of
smarts would be needed to also implement calltips or goto def/decl,
though perhaps with slightly different logic.

What's more, for the specific libclang case, since it's meant to be used
as a support library for IDEs (and other tools), it already provides a
function like `completeAt(where, ...)` for auto-completion that when
called provides a list of AST nodes referring to the valid completions
at that point. It would be horrible to have to re-implement that
function independent of the AST in TM/Geany, instead of just passing
Geany a list of strings or TM tags to show in the popup list.

Cheers,
Matthew Brush

[Jiří Techet]

Post by Thomas Martitz
Geany would then merge the tags, perhaps giving the plugin

Post by Thomas Martitz
ones more
weight, and store it in TM.
I think you underestimate how many tags we're talking here. The
example libclang ft-plugin would have to re-walk the entire AST
(which is absolutely massive, particularly for C++), convert it to
TM tag structures, and then Geany/TM would have to perform some
merging logic, would would be more complicated than now if it was
to support C++ properly, every single re-parse. My intuition tells
me that just won't be fast enough, Clang already jumps through
hoops and uses tricks to just build its own AST in-time.
I think it would be a disaster performance-wise. The number of AST nodes
can be easily 100x more than the amount of tags we have from ctags (we get
a single tag for a function now and AST will contain complete tree for the
function body) so just this might cost 100x more. In addition all the
necessary copies to TM internal representation, having to maintain the tree
structure (in TM we use GPtrArrays for everything which are very efficient
and during tag merge we try to eliminate even pointer dereferences because
those start getting expensive when managing many tags) etc.

Let's not outright reject possible solutions based on performance
assumptions and guestimations. Performance can be evaluated on an actual
implementation. Until then it's simply an invalid argument for this
discussion. But FWIW, I don't think performance is the driving aspect.

No, performance is a very valid point. Tag updates don't happen in a
background thread in Geany but rather on the main thread (and changing this
would require lots of modifications as neither ctags, nor TM nor Geany are
thread-safe) and all updates have to happen in a really short time period -
you cannot make the GUI freeze while the user is typing so you have 100ms
at most without any noticeable delay.

Post by Thomas Martitz
What's needed from the AST is tags (as you and I mentioned elsewhere, AST
is the complete code representation, so much more than what's required
currently). Those can be extracted in one pass. I don't see that tags need
to be converted back to the original AST.

As Matthew said, tags are insufficient for good autocompletion (e.g.
consider dynamic languages like Python where you have to infer variable
type from the right-hand side of an assignment and based on that generate
an autocompletion list).

Post by Thomas Martitz
For any successful plugin operation, the AST has to be generated (and
probably re-generated regularly) and traversed at least once. Creating tags
in a traversal that's happening done anyway probably isn't even going to
add much overhead, if any.

But I assume AST is created in a background thread - as I said, all TM code
runs on the main thread and everything has to happen fast enough between
keystrokes so users don't experience any delays while typing.

Post by Thomas Martitz
As you say, we use a GPtrArray of tags because it's very efficient for
sorting and merging. I think any ft-plugin will also have to sort (at
least) for showing auto-completion and symbols tree - if it shows them
itself. So it may even have to create such array/lists that TM uses anyway
(you can't sort AST directly). So it might as well pass them to TM for
sorting.

Whatever you do with tiny lists (say 1000 items) of tags which appear as a
result of autocompletion doesn't matter. I'm talking about workspace array
of tags which may contain all tags from a project with hundreds of
thousands or millions of tags and which have to be updated as the user
types and these updates have to be handled in a very efficient way.

Post by Thomas Martitz

Post by Thomas Martitz
And even if we did this, I don't know how we could handle ASTs of
different languages in a generic way because these will differ
significantly.

One more time, seems I wasn't clear enough yet: I'm *not* suggesting that
we create generic code inside Geany that handles any kind of AST. What I
suggest is that plugins that use AST internally (or not) pass tag-like
information to Geany, extracted and flattened from its internal AST (or
whatever it uses). Then Geany and other plugins can use that information
for their own purposes.

Again, for smart autocompletion you will need things from AST.

The biggest problem of current scope completion in Geany is the lack of
information about local variables which we might get with the new cxx ctags
parser and add them to TM. We'll get better results then but still
AST-based autocompletion will be able to do smarter things.

Cheers,

Jiri

[Thomas Martitz]

Post by Thomas Martitz
Geany would then merge the tags, perhaps giving the plugin
ones more
weight, and store it in TM.
I think you underestimate how many tags we're talking here. The
example libclang ft-plugin would have to re-walk the entire AST
(which is absolutely massive, particularly for C++), convert it to
TM tag structures, and then Geany/TM would have to perform some
merging logic, would would be more complicated than now if it was
to support C++ properly, every single re-parse. My
intuition tells
me that just won't be fast enough, Clang already jumps through
hoops and uses tricks to just build its own AST in-time.
I think it would be a disaster performance-wise. The number of
AST nodes can be easily 100x more than the amount of tags we
have from ctags (we get a single tag for a function now and
AST will contain complete tree for the function body) so just
this might cost 100x more. In addition all the necessary
copies to TM internal representation, having to maintain the
tree structure (in TM we use GPtrArrays for everything which
are very efficient and during tag merge we try to eliminate
even pointer dereferences because those start getting
expensive when managing many tags) etc.
Let's not outright reject possible solutions based on performance
assumptions and guestimations. Performance can be evaluated on an
actual implementation. Until then it's simply an invalid argument
for this discussion. But FWIW, I don't think performance is the
driving aspect.
No, performance is a very valid point. Tag updates don't happen in a
background thread in Geany but rather on the main thread (and changing
this would require lots of modifications as neither ctags, nor TM nor
Geany are thread-safe) and all updates have to happen in a really
short time period - you cannot make the GUI freeze while the user is
typing so you have 100ms at most without any noticeable delay.

I'm saying we can't evaluate performance at this time, because no
implementation is available. So saying now "uhm I fear it might be too
slow my guess is that the other one is 100x faster" is just a wild
assumption that doesn't help except spraying FUD. And outright rejecting
a proposal based on such assumptions is invalid.

Please lets evaluate solutions, implement them, and then have an eye on
performance.

Post by Thomas Martitz
What's needed from the AST is tags (as you and I mentioned
elsewhere, AST is the complete code representation, so much more
than what's required currently). Those can be extracted in one
pass. I don't see that tags need to be converted back to the
original AST.
As Matthew said, tags are insufficient for good autocompletion (e.g.
consider dynamic languages like Python where you have to infer
variable type from the right-hand side of an assignment and based on
that generate an autocompletion list).

Tags with incomplete information are insufficient. One solution can be
to have tags hold sufficient information.

FWIW, unless you actually compile the stuff (for C++), any source code
analysis will be insufficient for some cases. I was under the impression
that we do *not* want mandatory compilation, which drags in build system
implications, for stuff like auto completion.

But if a smart ft-plugin does this I don't care. I only care about
if/how Geany - and other non-ft-plugins - can use the data from such an
analysis.

Post by Thomas Martitz
For any successful plugin operation, the AST has to be generated
(and probably re-generated regularly) and traversed at least once.
Creating tags in a traversal that's happening done anyway probably
isn't even going to add much overhead, if any.
But I assume AST is created in a background thread - as I said, all TM
code runs on the main thread and everything has to happen fast enough
between keystrokes so users don't experience any delays while typing.

I don't care how the ft-plugin that provides tags to Geany generates
it's internal AST. It can surely use a background thread. This is quite
unrelated.

Post by Thomas Martitz
As you say, we use a GPtrArray of tags because it's very efficient
for sorting and merging. I think any ft-plugin will also have to
sort (at least) for showing auto-completion and symbols tree - if
it shows them itself. So it may even have to create such
array/lists that TM uses anyway (you can't sort AST directly). So
it might as well pass them to TM for sorting.
Whatever you do with tiny lists (say 1000 items) of tags which appear
as a result of autocompletion doesn't matter. I'm talking about
workspace array of tags which may contain all tags from a project with
hundreds of thousands or millions of tags and which have to be updated
as the user types and these updates have to be handled in a very
efficient way.

Wait, a AST comprising a million nodes doesn't result in a million
autocompletion candidates. Anyway, how does the number relate to our
discussion?

Post by Thomas Martitz
And even if we did this, I don't know how we could handle ASTs
of different languages in a generic way because these will
differ significantly.
One more time, seems I wasn't clear enough yet: I'm *not*
suggesting that we create generic code inside Geany that handles
any kind of AST. What I suggest is that plugins that use AST
internally (or not) pass tag-like information to Geany, extracted
and flattened from its internal AST (or whatever it uses). Then
Geany and other plugins can use that information for their own
purposes.
Again, for smart autocompletion you will need things from AST.
The biggest problem of current scope completion in Geany is the lack
of information about local variables which we might get with the new
cxx ctags parser and add them to TM. We'll get better results then but
still AST-based autocompletion will be able to do smarter things.

Again, using AST is fine, within the ft-plugin. Use whatever you need
from the AST for smart completion, and pass it in a generic format to
Geany. This way Geany *will use* AST-based information just fine.

Best regards.

[Colomban Wendling]

Post by Thomas Martitz

Post by Jiří Techet
[…]
No, performance is a very valid point. Tag updates don't happen in a
background thread in Geany but rather on the main thread (and changing
this would require lots of modifications as neither ctags, nor TM nor
Geany are thread-safe) and all updates have to happen in a really
short time period - you cannot make the GUI freeze while the user is
typing so you have 100ms at most without any noticeable delay.

I'm saying we can't evaluate performance at this time, because no
implementation is available. So saying now "uhm I fear it might be too
slow my guess is that the other one is 100x faster" is just a wild
assumption that doesn't help except spraying FUD. And outright rejecting
a proposal based on such assumptions is invalid.

I wouldn't dismiss Jiří's performances remarks that easily :) Remember
he spent a fair amount of time optimizing it, and has had experience
with huge amount of tags with his project-organizer plugin on large
projects (Linux kernel, anyone?).
And IIUC, he realized that with all the tags *current* parsers generate
(e.g. no local variables, etc.) TM started to struggle too much for it
to be usable.

Sure, we need numbers, but I'm afraid we kinda already have some idea of
what they would be.

Yes, maybe it'd be possible to improve the situation even further, but
it might be a lot of work for very little gain. I would guess if a e.g.
libclang works with real-size projects is because it has a lot of highly
specific optimizations resulting of a lot of work on their side. And
why I'm not sure it's even so useful for TM anyway, is that both Lex and
Matthew showed some semantic examples that are both very subtle to deal
with, and highly specific to some languages (here, C++) -- I love Lex's
ADL example, C++ can seem just crazy :)

Regards,
Colomban

[Lex Trotman]

I love Lex's ADL example, C++ can seem just crazy :)

Crazy like a Fox, makes the compiler near impossible, but makes lots
of stuff "just work" :)

So a much simpler example then (using C syntax to explain since we all
know it, but could be any language):

int a;
{
first_piece_of_code_using_a...
float a;
second_piece_of_code_using_a ....
}

For C/C++ the first piece of code will see `a` as an int and the
second piece of code will see `a` as a float, but there are languages
(Julia for one) where both pieces of code will see `a` as a float.
Again you need language specific knowledge to lookup `a` in the first
piece of code.

Cheers
Lex

Regards,
Colomban
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Thomas Martitz]

Post by Lex Trotman

I love Lex's ADL example, C++ can seem just crazy :)

Crazy like a Fox, makes the compiler near impossible, but makes lots
of stuff "just work" :)
So a much simpler example then (using C syntax to explain since we all
int a;
{
first_piece_of_code_using_a...
float a;
second_piece_of_code_using_a ....
}
For C/C++ the first piece of code will see `a` as an int and the
second piece of code will see `a` as a float, but there are languages
(Julia for one) where both pieces of code will see `a` as a float.
Again you need language specific knowledge to lookup `a` in the first
piece of code.

I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support"
all the time.

We just disagree on how the language-specific knowledge is transported
to Geany, other plugins and the user.

Best regards.

[Lex Trotman]

Post by Thomas Martitz
I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support" all
the time.
We just disagree on how the language-specific knowledge is transported to
Geany, other plugins and the user.

Well, I read your previous comments such as "But *just* the data,
don't offload very specific use cases onto them" (and to some extent
this one too) as saying that all the plugin has to do is return some
data (tags) and Geany will take it from there and all I was doing was
making sure it was clear that its unlikely that Geany will be able to
"take it from there" in any meaningful way, based purely on data,
without filetype specific knowledge inside Geany.

Geany must ask the plugin for the answer for each element of
functionality the plugin provides. This means that those elements
have indeed simply moved "Geany's deficiencies to the plugin space" as
you put it. This is good and necessary.

Of course moving the problem to plugin space doesn't mean the plugin
can't use Geany facilities where their capabilities fit the
requirement. But we should not try to expand Geany to handle all
types of functionality.

Like your TM query interface, the plugins should answer the questions
like "whats the autocomplete here", "what calltips are relevant here"
with a flat list of data relevant to the question.

The only place all the symbols should be returned is for the display
in the symbol tree, and for that use I'll defer to Colomban's
suggestion to provide TMTag structures, not for the plugin to access
TM.

Post by Thomas Martitz
Best regards.
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Thomas Martitz]

Post by Lex Trotman

Post by Thomas Martitz
I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support" all
the time.
We just disagree on how the language-specific knowledge is transported to
Geany, other plugins and the user.

Well, I read your previous comments such as "But *just* the data,
don't offload very specific use cases onto them" (and to some extent
this one too) as saying that all the plugin has to do is return some
data (tags) and Geany will take it from there and all I was doing was
making sure it was clear that its unlikely that Geany will be able to
"take it from there" in any meaningful way, based purely on data,
without filetype specific knowledge inside Geany.

Here we disagree. I think it is possible to extent TM/Geany sufficiently
to encode this information in generic ways. As an example, template
instantiation yields different types. TM could tread them as different
types. Any variable will be of one of the types, so calltips would just
work.

Post by Lex Trotman
Geany must ask the plugin for the answer for each element of
functionality the plugin provides. This means that those elements
have indeed simply moved "Geany's deficiencies to the plugin space" as
you put it. This is good and necessary.

To me this is not good because it makes it impossible to use the
information for other use cases in Geany or other plugins.

For example, I want to keep my quickswitch plugin that allows me to jump
to arbitrary tags, even if a ft-plugin is doing all the hard work to
parse them.

Post by Lex Trotman
Of course moving the problem to plugin space doesn't mean the plugin
can't use Geany facilities where their capabilities fit the
requirement. But we should not try to expand Geany to handle all
types of functionality.
Like your TM query interface, the plugins should answer the questions
like "whats the autocomplete here", "what calltips are relevant here"
with a flat list of data relevant to the question.

My TM query interface wants to return all matching tags, including those
found by ft-plugins. Can this be done?

Best regards

[Matthew Brush]

Post by Thomas Martitz

Post by Lex Trotman

Post by Thomas Martitz
I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support" all
the time.
We just disagree on how the language-specific knowledge is
transported to
Geany, other plugins and the user.

Well, I read your previous comments such as "But *just* the data,
don't offload very specific use cases onto them" (and to some extent
this one too) as saying that all the plugin has to do is return some
data (tags) and Geany will take it from there and all I was doing was
making sure it was clear that its unlikely that Geany will be able to
"take it from there" in any meaningful way, based purely on data,
without filetype specific knowledge inside Geany.

Here we disagree. I think it is possible to extent TM/Geany sufficiently
to encode this information in generic ways. As an example, template
instantiation yields different types. TM could tread them as different
types. Any variable will be of one of the types, so calltips would just
work.

Let's use the simplest C example:

gint x = g_...

Please describe the algorithm (just in prose) that would give only the
valid completions (ie. would compile in a C compiler) for `g_...`
without providing any impossible completions, and without knowing
anything about C/C++/Obj-C.

Post by Thomas Martitz

Post by Lex Trotman
[...]

My TM query interface wants to return all matching tags, including those
found by ft-plugins. Can this be done?

There's no reason an API couldn't be provided to query such information
even from ft-plugins, for ex.

void query_symbols(criteria, out_tags_list);

Geany doesn't have to know every tag in order to call into ft-plugins
for such query results, and can even fallback to TM/ctags if no plugins
support this feature.

Cheers,
Matthew Brush

[Matthew Brush]

Post by Matthew Brush

Post by Thomas Martitz

Post by Lex Trotman

Post by Thomas Martitz
I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support" all
the time.
We just disagree on how the language-specific knowledge is
transported to
Geany, other plugins and the user.

Well, I read your previous comments such as "But *just* the data,
don't offload very specific use cases onto them" (and to some extent
this one too) as saying that all the plugin has to do is return some
data (tags) and Geany will take it from there and all I was doing was
making sure it was clear that its unlikely that Geany will be able to
"take it from there" in any meaningful way, based purely on data,
without filetype specific knowledge inside Geany.

Here we disagree. I think it is possible to extent TM/Geany sufficiently
to encode this information in generic ways. As an example, template
instantiation yields different types. TM could tread them as different
types. Any variable will be of one of the types, so calltips would just
work.

gint x = g_...
Please describe the algorithm (just in prose) that would give only the
valid completions (ie. would compile in a C compiler) for `g_...`
without providing any impossible completions, and without knowing
anything about C/C++/Obj-C.

To make it more interesting, you can assume that the full C/C++/Obj-C
AST is somehow represented with full fidelity in a TM tags array, as a
given.

Cheers,
Matthew Brush

[Thomas Martitz]

Post by Matthew Brush

Post by Thomas Martitz

Post by Lex Trotman

Post by Thomas Martitz
I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support" all
the time.
We just disagree on how the language-specific knowledge is
transported to
Geany, other plugins and the user.

Well, I read your previous comments such as "But *just* the data,
don't offload very specific use cases onto them" (and to some extent
this one too) as saying that all the plugin has to do is return some
data (tags) and Geany will take it from there and all I was doing was
making sure it was clear that its unlikely that Geany will be able to
"take it from there" in any meaningful way, based purely on data,
without filetype specific knowledge inside Geany.

Here we disagree. I think it is possible to extent TM/Geany sufficiently
to encode this information in generic ways. As an example, template
instantiation yields different types. TM could tread them as different
types. Any variable will be of one of the types, so calltips would just
work.

gint x = g_...
Please describe the algorithm (just in prose) that would give only the
valid completions (ie. would compile in a C compiler) for `g_...`
without providing any impossible completions, and without knowing
anything about C/C++/Obj-C.

Do you suggest the completion list would only contain function that
return gint?

This is not what I would want. I can remember function names but not
necessarily their exact return value. It's possible (even likely) that I
got gint wrong and correct it to glong after auto completion, when the
call tip for the completed function tells me that glong. It'd be
annoying if auto-completion can't find the function I want just because
I misremembered the return type and the thing tries to be too smart.

So, as for the algorithm, I'd really stick to prefix matching, as done
currently. Everything else assumes that I coded everything correctly
which is not always the case. I don't want to be auto completion to be
too smart and hide wanted functions.

As Colomban and Jiri also mentioned it: auto-completion is also a useful
tool for correcting typos and other programming errors.

Post by Matthew Brush

Post by Thomas Martitz

Post by Lex Trotman
[...]

My TM query interface wants to return all matching tags, including those
found by ft-plugins. Can this be done?

There's no reason an API couldn't be provided to query such
information even from ft-plugins, for ex.
void query_symbols(criteria, out_tags_list);
Geany doesn't have to know every tag in order to call into ft-plugins
for such query results, and can even fallback to TM/ctags if no
plugins support this feature.

Okay. Might as well pass the tags to Geany in the first place but it's
important to make them available at all.

Best regards.

[Matthew Brush]

Post by Thomas Martitz

Post by Matthew Brush

Post by Thomas Martitz

Post by Lex Trotman

Post by Thomas Martitz
I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support" all
the time.
We just disagree on how the language-specific knowledge is
transported to
Geany, other plugins and the user.

Well, I read your previous comments such as "But *just* the data,
don't offload very specific use cases onto them" (and to some extent
this one too) as saying that all the plugin has to do is return some
data (tags) and Geany will take it from there and all I was doing was
making sure it was clear that its unlikely that Geany will be able to
"take it from there" in any meaningful way, based purely on data,
without filetype specific knowledge inside Geany.

Here we disagree. I think it is possible to extent TM/Geany sufficiently
to encode this information in generic ways. As an example, template
instantiation yields different types. TM could tread them as different
types. Any variable will be of one of the types, so calltips would just
work.

gint x = g_...
Please describe the algorithm (just in prose) that would give only the
valid completions (ie. would compile in a C compiler) for `g_...`
without providing any impossible completions, and without knowing
anything about C/C++/Obj-C.

Do you suggest the completion list would only contain function that
return gint?

Could be that, or it could be as simple as not listing things which
aren't in scope or visible (say an #include is missing). What I was
getting at is that unless you describe Geany's current algorithm, you'll
almost surely start describing C/C++/Obj-specific semantics encoded
inside Geany.

Post by Thomas Martitz
[...]
So, as for the algorithm, I'd really stick to prefix matching, as done
currently. Everything else assumes that I coded everything correctly
which is not always the case. I don't want to be auto completion to be
too smart and hide wanted functions.

Ok, I see where the problem is. You seem to only want the current
functionality while I want to enhance it to be more in-line with the
type of smarts you get in real IDEs that have deep knowledge of your
code and programming language. So if this is the case you're
fundamentally not going to like any features ft-plugins provide, and I
suggest that you just not use them. The point of adding all this in
plugins is specifically so you don't have to :)

Cheers,
Matthew Brush

[Lex Trotman]

Post by Lex Trotman

Post by Thomas Martitz
I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support" all
the time.
We just disagree on how the language-specific knowledge is transported to
Geany, other plugins and the user.

Well, I read your previous comments such as "But *just* the data,
don't offload very specific use cases onto them" (and to some extent
this one too) as saying that all the plugin has to do is return some
data (tags) and Geany will take it from there and all I was doing was
making sure it was clear that its unlikely that Geany will be able to
"take it from there" in any meaningful way, based purely on data,
without filetype specific knowledge inside Geany.

Here we disagree. I think it is possible to extent TM/Geany sufficiently to
encode this information in generic ways. As an example, template
instantiation yields different types. TM could tread them as different
types. Any variable will be of one of the types, so calltips would just
work.

It is of course always "possible", after all its a mere matter of programming :)

And of course an instantiated class template is just a type, but what
is the name of std::vector<int> and std::vector<A> so it can be looked
up?

Post by Lex Trotman
Geany must ask the plugin for the answer for each element of
functionality the plugin provides. This means that those elements
have indeed simply moved "Geany's deficiencies to the plugin space" as
you put it. This is good and necessary.

To me this is not good because it makes it impossible to use the information
for other use cases in Geany or other plugins.
For example, I want to keep my quickswitch plugin that allows me to jump to
arbitrary tags, even if a ft-plugin is doing all the hard work to parse
them.

Good point, the FT-Plugin API needs functions for goto declaration and
goto definition. Unlike TM, the plugin can take language and scope
rules into account when it answers, so you get the right declaration,
continuing the theme above, where would you go to find the declaration
of std::vector<int> ?

Certainly not the line:

std::vector<int> list_of_ints;

thats the declaration of `list_of_ints` but thats the only place that
`std::vector<int>` exists in the code.

Post by Lex Trotman
Of course moving the problem to plugin space doesn't mean the plugin
can't use Geany facilities where their capabilities fit the
requirement. But we should not try to expand Geany to handle all
types of functionality.
Like your TM query interface, the plugins should answer the questions
like "whats the autocomplete here", "what calltips are relevant here"
with a flat list of data relevant to the question.

My TM query interface wants to return all matching tags, including those
found by ft-plugins. Can this be done?

Only if your query plugin queries the FT plugin.

Best regards
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Thomas Martitz]

Post by Lex Trotman

Post by Lex Trotman

Post by Thomas Martitz
I think we all agree that help of language-specific plugins is
desired/required. No need to restate "we need language specific support" all
the time.
We just disagree on how the language-specific knowledge is transported to
Geany, other plugins and the user.

Well, I read your previous comments such as "But *just* the data,
don't offload very specific use cases onto them" (and to some extent
this one too) as saying that all the plugin has to do is return some
data (tags) and Geany will take it from there and all I was doing was
making sure it was clear that its unlikely that Geany will be able to
"take it from there" in any meaningful way, based purely on data,
without filetype specific knowledge inside Geany.

Here we disagree. I think it is possible to extent TM/Geany sufficiently to
encode this information in generic ways. As an example, template
instantiation yields different types. TM could tread them as different
types. Any variable will be of one of the types, so calltips would just
work.

It is of course always "possible", after all its a mere matter of programming :)
And of course an instantiated class template is just a type, but what
is the name of std::vector<int> and std::vector<A> so it can be looked
up?

vector<int> and vector<A>, with scope being std (I think trailing scope
delimiters are not saved in the scope string of TMTag, just intermediate
ones).

Post by Lex Trotman

Post by Lex Trotman
Geany must ask the plugin for the answer for each element of
functionality the plugin provides. This means that those elements
have indeed simply moved "Geany's deficiencies to the plugin space" as
you put it. This is good and necessary.

To me this is not good because it makes it impossible to use the information
for other use cases in Geany or other plugins.
For example, I want to keep my quickswitch plugin that allows me to jump to
arbitrary tags, even if a ft-plugin is doing all the hard work to parse
them.

Good point, the FT-Plugin API needs functions for goto declaration and
goto definition. Unlike TM, the plugin can take language and scope
rules into account when it answers, so you get the right declaration,
continuing the theme above, where would you go to find the declaration
of std::vector<int> ?
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that
`std::vector<int>` exists in the code.

Ideally the header which defines std::vector<A> would be found (and
maybe opened) right? Currently this isn't possible with TMTag (as
var_type can only contain either vector<A> or vector<int>) but it
shouldn't be too hard to make it work. Like adding a generic_type field
which is set for each template instance (containing vector<A>), then
instead of looking up the decl of generic_type if it's not NULL,
otherwise var_type.

Of course, the TMTags associated with vector<A> and vector<int> would
have to provided by the ft-plugin too (maybe there needs to be a new
TMTagType for vector<A> too).

Post by Lex Trotman

Post by Lex Trotman
Of course moving the problem to plugin space doesn't mean the plugin
can't use Geany facilities where their capabilities fit the
requirement. But we should not try to expand Geany to handle all
types of functionality.
Like your TM query interface, the plugins should answer the questions
like "whats the autocomplete here", "what calltips are relevant here"
with a flat list of data relevant to the question.

My TM query interface wants to return all matching tags, including those
found by ft-plugins. Can this be done?

Only if your query plugin queries the FT plugin.

So that means

1) My plugin contains code specifically for each known ft-plugin (i.e.
not the plugin API)
2) There needs to be a shared library which my plugin can use to call
into the other plugin
3) My plugin needs to know if and when to ask a ft-plugin

This is not acceptable. Or are you saying Geany could provide a wrapper
for this as Matthew suggested? That might be workable.

Best regards.

[Lex Trotman]

[...]

Post by Thomas Martitz

Post by Lex Trotman
It is of course always "possible", after all its a mere matter of programming :)
And of course an instantiated class template is just a type, but what
is the name of std::vector<int> and std::vector<A> so it can be looked
up?

vector<int> and vector<A>, with scope being std (I think trailing scope
delimiters are not saved in the scope string of TMTag, just intermediate
ones).

Sorry Thomas, I was a little mean and laid a trap.

std::vector has a second parameter, which has a default value so its
legal to omit it, but the type you will get from an accurate parser to
store in TM will be the complete type which will be something like
std::vector<int, std::allocator<int>> not just plain std::vector<int>.
This is the type you see printed in g++ and clang++ error messages for
template expansions, very messy.

So the way TM does a pure textual comparison of the prefix in the
source code `std::vector<int>` simply won't work to find
`std::vector<int,std::allocator<int>>`

It also won't work if there are spaces as I showed in a previous post
`std::vector< int >` is perfectly legal C++ but won't match
`std::vector<int>`. And before you ask, yes lots of code has the
spaces so that nested angle bracket closes (as in my
...allocator<int>> above) look like > > not the >> operator. Being
able to omit the space and not have it seen as a shift right operator
is only a recent addition to the standard.

And for template functions declared as `template<class T>T& f(T& t){
return t<<1; }` they are used as simply `int a = f(1);` not an angle
bracket in sight, and a textual comparison to find the declaration for
a calltip isn't going to work because no such explicit declaration
exists. So how can TM answer whats legal here `f(std::cout).` (answer
all of std::ostream) versus here `f(1).` (answer nothing).

[...]

Post by Thomas Martitz

Post by Lex Trotman
Good point, the FT-Plugin API needs functions for goto declaration and
goto definition. Unlike TM, the plugin can take language and scope
rules into account when it answers, so you get the right declaration,
continuing the theme above, where would you go to find the declaration
of std::vector<int> ?
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that
`std::vector<int>` exists in the code.

Ideally the header which defines std::vector<A> would be found (and maybe
opened) right? Currently this isn't possible with TMTag (as var_type can
only contain either vector<A> or vector<int>) but it shouldn't be too hard
to make it work.

But now you are encoding language specific semantics into TM, and as I
noted above its more complicated than just that example.

Please don't get the idea from simple examples posted here that they
cover even a fraction of the craziness of languages (and not just C++
either, its simply the one I'm most familiar with).

Like adding a generic_type field which is set for each

Post by Thomas Martitz
template instance (containing vector<A>), then instead of looking up the
decl of generic_type if it's not NULL, otherwise var_type.
Of course, the TMTags associated with vector<A> and vector<int> would have
to provided by the ft-plugin too (maybe there needs to be a new TMTagType
for vector<A> too).

And don't forget that when trying to look up
`std::vector<int>::value_type` to see whats next, or to find the
declaration, either Geany or TM has to parse it and break it into the
elements `std`, `vector<int>` and `value_type`, ie has to know the
language syntax for template specialisations and namespaces,
`vector<int>` is not a legal name. And as I said before, that syntax
varies between languages, so again you are encoding language specifics
into Geany or TM.

And in a final real piece of evil C++ for the day, variadic templates
with variable numbers of parameters (like C variadic functions with
...).

std::get<1>(std::make_tuple(1 ,std::string("abc"), 3)).size(

ok, lets lookup the return type of std::make_tuple
http://en.cppreference.com/w/cpp/utility/tuple/make_tuple thats
simple-ish
and the type of std::get<1>
http://en.cppreference.com/w/cpp/utility/tuple/get hmmmm

And yep its all completely legal and idiomatic C++ and totally static
typed so should be no problem analysing it.

Sorry Colomban you didn't want see half the craziness of C++ I know :)

Post by Thomas Martitz

Post by Lex Trotman

Post by Thomas Martitz

Post by Lex Trotman
Of course moving the problem to plugin space doesn't mean the plugin
can't use Geany facilities where their capabilities fit the
requirement. But we should not try to expand Geany to handle all
types of functionality.
Like your TM query interface, the plugins should answer the questions
like "whats the autocomplete here", "what calltips are relevant here"
with a flat list of data relevant to the question.

My TM query interface wants to return all matching tags, including those
found by ft-plugins. Can this be done?

Only if your query plugin queries the FT plugin.

So that means
1) My plugin contains code specifically for each known ft-plugin (i.e. not
the plugin API)

No, but the implementation inside Geany of the query interface that
you have added in your PR may have to know to call the FT-plugin to
answer the query, but the plugin using the query interface doesn't
need to know about that.

Post by Thomas Martitz
2) There needs to be a shared library which my plugin can use to call into
the other plugin

I'm not sure why you would need a third dll even if you were having to
call the FT-plugin directly, you probably have to be passed its
g_module though. But you shouldn't have to access the FT-plugin
directly unless you are doing something Geany doesn't support.

Post by Thomas Martitz
3) My plugin needs to know if and when to ask a ft-plugin

As I said unless you are asking a question Geany doesn't know how to
answer, then you shouldn't need to.

Post by Thomas Martitz
This is not acceptable.

Be aware that as you say that, the alternative is for you to implement
all those language specifics I mentioned above in your plugin itself.

Post by Thomas Martitz
Or are you saying Geany could provide a wrapper for
this as Matthew suggested? That might be workable.

Yes, exactly. As Matthew said in a previous post, for each feature
where a FT-plugin is involved the function inside Geany that provides
that feature is going to have to have an if near the front that makes
the decision if it uses the plugin or does it itself. So if your
plugin uses that function you get the benefits of both
implementations.

Cheers
Lex

Post by Thomas Martitz
Best regards.
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Thomas Martitz]

Post by Lex Trotman
[...]

Post by Thomas Martitz

Post by Lex Trotman
It is of course always "possible", after all its a mere matter of programming :)
And of course an instantiated class template is just a type, but what
is the name of std::vector<int> and std::vector<A> so it can be looked
up?

vector<int> and vector<A>, with scope being std (I think trailing scope
delimiters are not saved in the scope string of TMTag, just intermediate
ones).

Sorry Thomas, I was a little mean and laid a trap.

No problem. The devil is always in the details and we can only define
requirements when looking and complicated cases.

Post by Lex Trotman
std::vector has a second parameter, which has a default value so its
legal to omit it, but the type you will get from an accurate parser to
store in TM will be the complete type which will be something like
std::vector<int, std::allocator<int>> not just plain std::vector<int>.
This is the type you see printed in g++ and clang++ error messages for
template expansions, very messy.
So the way TM does a pure textual comparison of the prefix in the
source code `std::vector<int>` simply won't work to find
`std::vector<int,std::allocator<int>>`
It also won't work if there are spaces as I showed in a previous post
`std::vector< int >` is perfectly legal C++ but won't match
`std::vector<int>`. And before you ask, yes lots of code has the
spaces so that nested angle bracket closes (as in my
...allocator<int>> above) look like > > not the >> operator. Being
able to omit the space and not have it seen as a shift right operator
is only a recent addition to the standard.

OK, so we'd have to have a canonical representation of types (how this
looks like is defined by the ft-plugin, e.g. whitespaces stripped,
default parameters expanded) to compare against, and a method to find
that tag when looking up std::vector< int >. It seems be complicated
indeed, but doable. Most of the complexity will be in the ft-plugin
providing canonicalized, tag-like data.

Post by Lex Trotman
And for template functions declared as `template<class T>T& f(T& t){
return t<<1; }` they are used as simply `int a = f(1);` not an angle
bracket in sight, and a textual comparison to find the declaration for
a calltip isn't going to work because no such explicit declaration
exists. So how can TM answer whats legal here `f(std::cout).` (answer
all of std::ostream) versus here `f(1).` (answer nothing).

I would say that a calltip for a templated function should show the
generic function since you don't know what the user is going to type,
regardless of the design (except perhaps if the specialized function can
be inferred from the scope). Which specialized function is to be called
changes as the user types. As I said in the other mail to Matthew,
trying to infer from the lhs of the assignment is dangerous and
impractical too (I would not want this behavior), especially if the lhs
is determined only by the rhs (auto x = foo(1)).

Perhaps you can show specialized calltips once user has entered a few
parameters but that's still guessing. I don't think this is a good idea.

But plain textual comparison against the function name seems to work
best here to me.

Post by Lex Trotman
[...]

Post by Thomas Martitz

Post by Lex Trotman
Good point, the FT-Plugin API needs functions for goto declaration and
goto definition. Unlike TM, the plugin can take language and scope
rules into account when it answers, so you get the right declaration,
continuing the theme above, where would you go to find the declaration
of std::vector<int> ?
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that
`std::vector<int>` exists in the code.

Ideally the header which defines std::vector<A> would be found (and maybe
opened) right? Currently this isn't possible with TMTag (as var_type can
only contain either vector<A> or vector<int>) but it shouldn't be too hard
to make it work.

But now you are encoding language specific semantics into TM, and as I
noted above its more complicated than just that example.

Generic ways to deal with language specific properties isn't quite the
same. Also, this doesn't apply to just C++, other languages have complex
meta programming / generic types too (Vala, Java, C#, etc.).

And even then, I'm totally fine with TM knowing language specifics. TM
can hold every information needed to make smart language specific
features work, preferably in generic ways. That doesn't mean TM has to
semantically understand the language's source code, if the information
is provided by external parsers. In fact, it doesn't have to understand
source code itself at all.

There may be a point we want to limit the smartness anyway, if requires
very complex solutions for little gain, extremely rare use cases or they
it can become annoying to the user (e.g. if it requires to much setup).

Post by Lex Trotman
Please don't get the idea from simple examples posted here that they
cover even a fraction of the craziness of languages (and not just C++
either, its simply the one I'm most familiar with).
Like adding a generic_type field which is set for each

Post by Thomas Martitz
template instance (containing vector<A>), then instead of looking up the
decl of generic_type if it's not NULL, otherwise var_type.
Of course, the TMTags associated with vector<A> and vector<int> would have
to provided by the ft-plugin too (maybe there needs to be a new TMTagType
for vector<A> too).

And don't forget that when trying to look up
`std::vector<int>::value_type` to see whats next, or to find the
declaration, either Geany or TM has to parse it and break it into the
elements `std`, `vector<int>` and `value_type`, ie has to know the
language syntax for template specialisations and namespaces,
`vector<int>` is not a legal name. And as I said before, that syntax
varies between languages, so again you are encoding language specifics
into Geany or TM.

I don't understand that. TM doesn't need to know that. It just has some
strings contained in tags that are matched against other strings. TM
doesn't break A::B::X, the parser does (currently this is split into
A::B and X). TM already has the concept of scope for that. It works as
of now, without knowing langauge specific syntax.

Post by Lex Trotman
And in a final real piece of evil C++ for the day, variadic templates
with variable numbers of parameters (like C variadic functions with
...).
std::get<1>(std::make_tuple(1 ,std::string("abc"), 3)).size(
ok, lets lookup the return type of std::make_tuple
http://en.cppreference.com/w/cpp/utility/tuple/make_tuple thats
simple-ish
and the type of std::get<1>
http://en.cppreference.com/w/cpp/utility/tuple/get hmmmm
And yep its all completely legal and idiomatic C++ and totally static
typed so should be no problem analysing it.
Sorry Colomban you didn't want see half the craziness of C++ I know :)

I think we the above suggestions this should work. Sure it must be
parsed by a ft-plugin.

You're heavy on templates which require compilation. I'm interested to
see how you want realize this (passing CFLAGS etc.) in a user friendly
way on a per-ft-plugin basis.

Post by Lex Trotman

Post by Thomas Martitz

Post by Lex Trotman

Post by Thomas Martitz

Post by Lex Trotman
Of course moving the problem to plugin space doesn't mean the plugin
can't use Geany facilities where their capabilities fit the
requirement. But we should not try to expand Geany to handle all
types of functionality.
Like your TM query interface, the plugins should answer the questions
like "whats the autocomplete here", "what calltips are relevant here"
with a flat list of data relevant to the question.

My TM query interface wants to return all matching tags, including those
found by ft-plugins. Can this be done?

Only if your query plugin queries the FT plugin.

So that means
1) My plugin contains code specifically for each known ft-plugin (i.e. not
the plugin API)

No, but the implementation inside Geany of the query interface that
you have added in your PR may have to know to call the FT-plugin to
answer the query, but the plugin using the query interface doesn't
need to know about that.

Okay. That's at least one more good reason to have the query interface :-)

Best regards

[Lex Trotman]

[...]>>

Post by Lex Trotman
std::vector has a second parameter, which has a default value so its
legal to omit it, but the type you will get from an accurate parser to
store in TM will be the complete type which will be something like
std::vector<int, std::allocator<int>> not just plain std::vector<int>.
This is the type you see printed in g++ and clang++ error messages for
template expansions, very messy.
So the way TM does a pure textual comparison of the prefix in the
source code `std::vector<int>` simply won't work to find
`std::vector<int,std::allocator<int>>`
It also won't work if there are spaces as I showed in a previous post
`std::vector< int >` is perfectly legal C++ but won't match
`std::vector<int>`. And before you ask, yes lots of code has the
spaces so that nested angle bracket closes (as in my
...allocator<int>> above) look like > > not the >> operator. Being
able to omit the space and not have it seen as a shift right operator
is only a recent addition to the standard.

OK, so we'd have to have a canonical representation of types (how this looks
like is defined by the ft-plugin, e.g. whitespaces stripped, default
parameters expanded) to compare against, and a method to find that tag when
looking up std::vector< int >. It seems be complicated indeed, but doable.
Most of the complexity will be in the ft-plugin providing canonicalized,
tag-like data.

There is no guarantee that a gcc based library (assuming RMS allows it
one day) would use the same canonical form as libclang, in fact it
probably deliberately would not, so its hard to swap them, so now you
not only have language specific code in Geany, but FT-plugin specific
code. And for other languages FT-plugins too.

And it still doesn't address the default template parameter issue above.

Post by Lex Trotman
And for template functions declared as `template<class T>T& f(T& t){
return t<<1; }` they are used as simply `int a = f(1);` not an angle
bracket in sight, and a textual comparison to find the declaration for
a calltip isn't going to work because no such explicit declaration
exists. So how can TM answer whats legal here `f(std::cout).` (answer
all of std::ostream) versus here `f(1).` (answer nothing).

I would say that a calltip for a templated function should show the generic
function since you don't know what the user is going to type, regardless of
the design (except perhaps if the specialized function can be inferred from
the scope). Which specialized function is to be called changes as the user
types. As I said in the other mail to Matthew, trying to infer from the lhs
of the assignment is dangerous and impractical too (I would not want this
behavior), especially if the lhs is determined only by the rhs (auto x =
foo(1)).

Yes, in C++ you can't infer from left to right, but there are
languages where this happens, Haskel, O'Caml that I immediately think
of. But my example was just to point out that the template parameter
is inferred from the type of the function parameter, the user does not
specify `f<int>(1)` sorry for the confusion.

And yes, as I showed in another post, accurate calltips can change as
parameters are added.

But the last two examples above are not calltips, they are
autocomplete, the function call is finished.

Both end in a dot accessing members of the return from the function.
To get accurate type info it is necessary to first recognise that a
template is needed for `f` (there are no <>s in the expressions) then
recognise that its return type matches its parameter type, then you
can suggest what members can come after the dot.

And just to note neither the return or the function parameter need to
be exactly the template parameter, for example `template<class T> T&
f(T* a)` will happily deduce a `T` from a `T*` and make the return
type `T&`. And several other type expressions work.

Perhaps you can show specialized calltips once user has entered a few
parameters but that's still guessing. I don't think this is a good idea.

But the FT-plugin may do this (at least IIUC libclang will) so its
fine if Geany doesn't, but it has to ask the FT-plugin not use TM.

But plain textual comparison against the function name seems to work best
here to me.

Depends what you are coding, as the car manufacturers say, "your
mileage may vary".

Post by Lex Trotman
[...]

Post by Thomas Martitz

Post by Lex Trotman
Good point, the FT-Plugin API needs functions for goto declaration and
goto definition. Unlike TM, the plugin can take language and scope
rules into account when it answers, so you get the right declaration,
continuing the theme above, where would you go to find the declaration
of std::vector<int> ?
std::vector<int> list_of_ints;
thats the declaration of `list_of_ints` but thats the only place that
`std::vector<int>` exists in the code.

Ideally the header which defines std::vector<A> would be found (and maybe
opened) right? Currently this isn't possible with TMTag (as var_type can
only contain either vector<A> or vector<int>) but it shouldn't be too hard
to make it work.

But now you are encoding language specific semantics into TM, and as I
noted above its more complicated than just that example.

Generic ways to deal with language specific properties isn't quite the same.
Also, this doesn't apply to just C++, other languages have complex meta
programming / generic types too (Vala, Java, C#, etc.).
And even then, I'm totally fine with TM knowing language specifics.

I keep saying Geany as well, not just TM, the language specifics to
storing stuff is one thing, the language specifics for using that
stored stuff is another more complex thing. And, if you have to ask
the plugin for the more complex questions, there is no point of
storing all the gory details in TM, because the plugin isn't going to
use it.

TM can
hold every information needed to make smart language specific features work,
preferably in generic ways. That doesn't mean TM has to semantically
understand the language's source code, if the information is provided by
external parsers. In fact, it doesn't have to understand source code itself
at all.

So as originally proposed, Geany has to ask the plugin first.

There may be a point we want to limit the smartness anyway, if requires very
complex solutions for little gain, extremely rare use cases or they it can
become annoying to the user (e.g. if it requires to much setup).

To emphasise I am not saying don't make TM and Geany smarter, but I am
saying don't make a `c.c` like mess of language specific code in Geany
or TM, and don't say "this is good enough for me so it should be good
enough for you" and prevent the use of smarter plugins for those who
want them. That means the plugin has to bypass the code in TM/Geany
to bypass its restrictions.

Post by Lex Trotman
Please don't get the idea from simple examples posted here that they
cover even a fraction of the craziness of languages (and not just C++
either, its simply the one I'm most familiar with).
Like adding a generic_type field which is set for each

Post by Thomas Martitz
template instance (containing vector<A>), then instead of looking up the
decl of generic_type if it's not NULL, otherwise var_type.
Of course, the TMTags associated with vector<A> and vector<int> would have
to provided by the ft-plugin too (maybe there needs to be a new TMTagType
for vector<A> too).

And don't forget that when trying to look up
`std::vector<int>::value_type` to see whats next, or to find the
declaration, either Geany or TM has to parse it and break it into the
elements `std`, `vector<int>` and `value_type`, ie has to know the
language syntax for template specialisations and namespaces,
`vector<int>` is not a legal name. And as I said before, that syntax
varies between languages, so again you are encoding language specifics
into Geany or TM.

I don't understand that. TM doesn't need to know that. It just has some
strings contained in tags that are matched against other strings. TM doesn't
break A::B::X, the parser does (currently this is split into A::B and X). TM
already has the concept of scope for that. It works as of now, without
knowing langauge specific syntax.

ATM Geany does that breakup of the expression in the source code and
asks TM for each element.

As I said above, if the plugin parser is doing it, the plugin parser
will look it up in its accurate language specific symbol table as
well, it won't use TM. So no need to store language specifics in TM
if you are using such a library.

To go back to one of your very initial comments "don't want to move
everything from TM/Geany to the plugin" my intention in all these
examples is to show that an accurate plugin MUST bypass TM/Geany
because they will always have limitations without effectively having
the same code as the plugin, ie a compiler.

Post by Lex Trotman
And in a final real piece of evil C++ for the day, variadic templates
with variable numbers of parameters (like C variadic functions with
...).
std::get<1>(std::make_tuple(1 ,std::string("abc"), 3)).size(
ok, lets lookup the return type of std::make_tuple
http://en.cppreference.com/w/cpp/utility/tuple/make_tuple thats
simple-ish
and the type of std::get<1>
http://en.cppreference.com/w/cpp/utility/tuple/get hmmmm
And yep its all completely legal and idiomatic C++ and totally static
typed so should be no problem analysing it.
Sorry Colomban you didn't want see half the craziness of C++ I know :)

I think we the above suggestions this should work. Sure it must be parsed by
a ft-plugin.
You're heavy on templates which require compilation. I'm interested to see
how you want realize this (passing CFLAGS etc.) in a user friendly way on a
per-ft-plugin basis.

Yes, to get the benefit of the plugins wisdom, the price is that it
needs to know how the software is to be built (at least for libclang).
But thats the plugins problem, not Geanys. Its a trade off. It might
not be a worthwhile trade off for you, but don't disallow usage by
others for whom the tradeoff is different. And as its a plugin you
can decide to not use it.

Post by Lex Trotman

Post by Thomas Martitz

Post by Lex Trotman

Post by Thomas Martitz

Post by Lex Trotman
Of course moving the problem to plugin space doesn't mean the plugin
can't use Geany facilities where their capabilities fit the
requirement. But we should not try to expand Geany to handle all
types of functionality.
Like your TM query interface, the plugins should answer the questions
like "whats the autocomplete here", "what calltips are relevant here"
with a flat list of data relevant to the question.

My TM query interface wants to return all matching tags, including those
found by ft-plugins. Can this be done?

Only if your query plugin queries the FT plugin.

So that means
1) My plugin contains code specifically for each known ft-plugin (i.e. not
the plugin API)

No, but the implementation inside Geany of the query interface that
you have added in your PR may have to know to call the FT-plugin to
answer the query, but the plugin using the query interface doesn't
need to know about that.

Okay. That's at least one more good reason to have the query interface :-)

I havn't looked in detail at your query interface, but the approach of
providing general wrapper APIs for external use, instead of exposing
the guts of Geany, I very much support. Remember waaaay back when we
were doing the build system, I proposed a set of functions to expose
it to the plugin interface in a generic manner, but they were vetoed
by Nick.


Cheers
Lex

Best regards
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Jiří Techet]

Post by Thomas Martitz

Post by Thomas Martitz
Geany would then merge the tags, perhaps giving the plugin
ones more
weight, and store it in TM.
I think you underestimate how many tags we're talking here. The
example libclang ft-plugin would have to re-walk the entire AST
(which is absolutely massive, particularly for C++), convert it to
TM tag structures, and then Geany/TM would have to perform some
merging logic, would would be more complicated than now if it was
to support C++ properly, every single re-parse. My intuition tells
me that just won't be fast enough, Clang already jumps through
hoops and uses tricks to just build its own AST in-time.
I think it would be a disaster performance-wise. The number of
AST nodes can be easily 100x more than the amount of tags we
have from ctags (we get a single tag for a function now and
AST will contain complete tree for the function body) so just
this might cost 100x more. In addition all the necessary
copies to TM internal representation, having to maintain the
tree structure (in TM we use GPtrArrays for everything which
are very efficient and during tag merge we try to eliminate
even pointer dereferences because those start getting
expensive when managing many tags) etc.
Let's not outright reject possible solutions based on performance
assumptions and guestimations. Performance can be evaluated on an
actual implementation. Until then it's simply an invalid argument
for this discussion. But FWIW, I don't think performance is the
driving aspect.
No, performance is a very valid point. Tag updates don't happen in a
background thread in Geany but rather on the main thread (and changing this
would require lots of modifications as neither ctags, nor TM nor Geany are
thread-safe) and all updates have to happen in a really short time period -
you cannot make the GUI freeze while the user is typing so you have 100ms
at most without any noticeable delay.

I'm saying we can't evaluate performance at this time, because no
implementation is available. So saying now "uhm I fear it might be too slow
my guess is that the other one is 100x faster" is just a wild assumption
that doesn't help except spraying FUD. And outright rejecting a proposal
based on such assumptions is invalid.

100x more nodes of AST than the amount of tags we have isn't any wild
assumption - have a look at the AST picture here:

https://en.wikipedia.org/wiki/Abstract_syntax_tree

and the primitive code below to which it corresponds. The tree has 21
nodes. It's easy to imagine source files with 5x more code per function and
you are at the numbers I'm giving you - we generate 1 tag per function and
the corresponding AST you'll have to walk is 100x bigger.

Post by Thomas Martitz
Please lets evaluate solutions, implement them, and then have an eye on
performance.

I think I have evaluated this solution and it doesn't seem worth the
effort. Not only because the performance but also I don't know how to store
all the necessary information for any kind of language into TM so code
completion works for any language so that it's compatible with ctags and
any kind of AST. If you do, please tell us but not the way "One solution
can be to have tags hold sufficient information" because it doesn't tell
anything.

Post by Thomas Martitz

Post by Thomas Martitz
What's needed from the AST is tags (as you and I mentioned
elsewhere, AST is the complete code representation, so much more
than what's required currently). Those can be extracted in one
pass. I don't see that tags need to be converted back to the
original AST.
As Matthew said, tags are insufficient for good autocompletion (e.g.
consider dynamic languages like Python where you have to infer variable
type from the right-hand side of an assignment and based on that generate
an autocompletion list).

Tags with incomplete information are insufficient. One solution can be to
have tags hold sufficient information.
FWIW, unless you actually compile the stuff (for C++), any source code
analysis will be insufficient for some cases. I was under the impression
that we do *not* want mandatory compilation, which drags in build system
implications, for stuff like auto completion.

libclang basically compiles it - it performs complete syntax analysis the
same way that would happen during compilation. It just skips code
generation.

Post by Thomas Martitz
But if a smart ft-plugin does this I don't care. I only care about if/how
Geany - and other non-ft-plugins - can use the data from such an analysis.

Post by Thomas Martitz
For any successful plugin operation, the AST has to be generated
(and probably re-generated regularly) and traversed at least once.
Creating tags in a traversal that's happening done anyway probably
isn't even going to add much overhead, if any.
But I assume AST is created in a background thread - as I said, all TM
code runs on the main thread and everything has to happen fast enough
between keystrokes so users don't experience any delays while typing.

I don't care how the ft-plugin that provides tags to Geany generates it's
internal AST. It can surely use a background thread. This is quite
unrelated.

I was reacting to you where you said that AST has to be regenerated
regularly and that time spent on updating TM will be smaller. Yes, it will
be smaller, but it will have to be performed on the main thread instead of
in the background so the performance will be much more critical.

Post by Thomas Martitz

Post by Thomas Martitz
As you say, we use a GPtrArray of tags because it's very efficient
for sorting and merging. I think any ft-plugin will also have to
sort (at least) for showing auto-completion and symbols tree - if
it shows them itself. So it may even have to create such
array/lists that TM uses anyway (you can't sort AST directly). So
it might as well pass them to TM for sorting.
Whatever you do with tiny lists (say 1000 items) of tags which appear as
a result of autocompletion doesn't matter. I'm talking about workspace
array of tags which may contain all tags from a project with hundreds of
thousands or millions of tags and which have to be updated as the user
types and these updates have to be handled in a very efficient way.

Wait, a AST comprising a million nodes doesn't result in a million
autocompletion candidates. Anyway, how does the number relate to our
discussion?

It appears to me you don't really read what I'm writing - I'm talking about
updating TM workspace tags which has to happen every time file is reparsed.
As I said, autocompletion list population performance doesn't matter much
because it will contain a lot less tags.

And by the way, projects with 1 000 000 tags will have more like 100 000
000 nodes in AST.

Post by Thomas Martitz

Post by Thomas Martitz
And even if we did this, I don't know how we could handle ASTs
of different languages in a generic way because these will
differ significantly.
One more time, seems I wasn't clear enough yet: I'm *not*
suggesting that we create generic code inside Geany that handles
any kind of AST. What I suggest is that plugins that use AST
internally (or not) pass tag-like information to Geany, extracted
and flattened from its internal AST (or whatever it uses). Then
Geany and other plugins can use that information for their own
purposes.
Again, for smart autocompletion you will need things from AST.
The biggest problem of current scope completion in Geany is the lack of
information about local variables which we might get with the new cxx ctags
parser and add them to TM. We'll get better results then but still
AST-based autocompletion will be able to do smarter things.

Again, using AST is fine, within the ft-plugin. Use whatever you need from
the AST for smart completion, and pass it in a generic format to Geany.
This way Geany *will use* AST-based information just fine.

I'm ending the discussion here, don't expect more contributions on this
topic from me. You don't listen to what others are saying and keep
rejecting their arguments without giving any concrete proposals other than
"Use whatever you need" and "and pass it in a generic format to Geany"
which tell nothing. Continuing in this is a waste of time for me.

Jiri

[Matthew Brush]

Post by Jiří Techet
[...]

Post by Thomas Martitz
Wait, a AST comprising a million nodes doesn't result in a million
autocompletion candidates. Anyway, how does the number relate to our
discussion?

It appears to me you don't really read what I'm writing - I'm talking about
updating TM workspace tags which has to happen every time file is reparsed.
As I said, autocompletion list population performance doesn't matter much
because it will contain a lot less tags.
And by the way, projects with 1 000 000 tags will have more like 100 000
000 nodes in AST.

To give a sensational example, a minimal "hello world" C++ program:

#include <iostream>
int main()
{
std::cout << "Hello World" << std::endl;
return 0;
}

CTags:

$ ctags -o - hello-world.cc | wc -l
1

Clang AST:

$ clang++ -Xclang -ast-dump -fno-diagnostics-color \
-fsyntax-only hello-world.cc | wc -l
37806

Obviously since C++ compilers operate on translation units, the latter
includes all of what's brought in through includes. If I manually strip
it down to just what humans see in the source file, it's around 15
lines, but the 37806 is what a libclang ft-plugin would be working with
in the AST.

I couldn't speculate what a larger project like LLVM itself would parse
to, but I suspect it would take a not-insignificant amount of time for
an ft-plugin to even traverse the AST once (eg. to find all the nodes
originating in a particular source file).

Cheers,
Matthew Brush

[Lex Trotman]

This adds per use case hooks to plugins, which then became part of the
stable API. I don't think that we have to codify every single use case of
tags into the plugins. That's just making it harder (maybe impossible?) to
change or add use cases.
The point of this proposal is to change and add use-cases that are not
currently possible with the current plugin API. But instead of each
use-case generating its own piece of API and its own infrastructure,
the point of the FT-plugins proposal is to provide a common
infrastructure and approach for all filetype specific use-cases, those
needed for currently suggested uses, indentation, clang based styling
and symbols, and as framework for future use-cases we either havn't
thought of, or havn't a concrete intention to add immediately.

Post by Thomas Martitz
I thought we agreed that plugin should simply provide tags to Geany/TM

This proposal is about many types of filetype specific functionality,
not just tags. Tagmanager will not help in any way with indenting
Haskell, or even C++.

4 of 5 of the proposed features are strictly tag-related. And Geany can do
all of them already, it's just that the current implementation leaves things
to be desired so there is the idea to let plugins improve upon them.

Well, 3 out of 6 but whos counting :)

Certainly 1) showing symbols in the symbol list, 2) autocomplete and
3) calltips are currently available to a degree in Geany. But
highlighting, build commands and build result handling are not. But
to be able to do 2) and 3) accurately needs more knowledge of each
language semantics than is currently available in Geany or tagmanager.

I disagree with the proposed solution for those 4, because they are
offloading logic on a per feature basis to plugins, only because Geany isn't
capable at the moment. If Geany was capable, then there could be 1 solution
for the 4 features and less complexity in each plugin (and we know the
quality of plugins varies a lot so they should have little complexity as
possible).

Encoding the knowledge of language semantics into Geany, for each
language supported, is going to make autocomplete and calltip code
look like c.c. Its not the way to go.

The solution I have in mind simply allows plugins to pass tags to Geany
which they parsed with more advanced code. The tags itself would advanced
too, to allow for the improvements current TM+ctags can't offer. Symbol
tree, calltips, autocompletion, jump-to-decl can all be improved based on
the advanced tags.

Well, again you are encoding language semantics into Geany, for
example for C++ that means autocompletion and calltips need to handle
1) local symbol scopes, 2) member functions being in the scope of the
class, even when they are not 3) argument dependent lookup 4) template
expansion lookup and 5) handling of template parameter based typing.
These are hard, just ask the GCC and clang guys. And every user of
Geany will have to pay the cost of the code they don't use, unless
they use C++.

Then for a multidispatch language like Julia you need to handle
overloading in an even more subtle way than C++ overloading.

And why re-implement these language specific subtle and difficult
features in Geany when more and more languages are providing libclang
like libraries to do it for us, accurately and up to date with the
language.

Cheers
Lex

Best regards.
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Thomas Martitz]

Post by Lex Trotman

This adds per use case hooks to plugins, which then became part of the
stable API. I don't think that we have to codify every single use case of
tags into the plugins. That's just making it harder (maybe impossible?) to
change or add use cases.
The point of this proposal is to change and add use-cases that are not
currently possible with the current plugin API. But instead of each
use-case generating its own piece of API and its own infrastructure,
the point of the FT-plugins proposal is to provide a common
infrastructure and approach for all filetype specific use-cases, those
needed for currently suggested uses, indentation, clang based styling
and symbols, and as framework for future use-cases we either havn't
thought of, or havn't a concrete intention to add immediately.

Post by Thomas Martitz
I thought we agreed that plugin should simply provide tags to Geany/TM

This proposal is about many types of filetype specific functionality,
not just tags. Tagmanager will not help in any way with indenting
Haskell, or even C++.

4 of 5 of the proposed features are strictly tag-related. And Geany can do
all of them already, it's just that the current implementation leaves things
to be desired so there is the idea to let plugins improve upon them.

Well, 3 out of 6 but whos counting :)

I count "Go To Declaration/Definition" as a tag issue. Where a symbol is
stored,
and whether it's a definition or declaration, is contained in tags.

I did not count the build/run support, diagnostics and refactoring
because there was no specific propsal given.

Post by Lex Trotman
Certainly 1) showing symbols in the symbol list, 2) autocomplete and
3) calltips are currently available to a degree in Geany. But
highlighting, build commands and build result handling are not.
But
to be able to do 2) and 3) accurately needs more knowledge of each
language semantics than is currently available in Geany or tagmanager.

That's right. But it doesn't mean the features should be *entirely*
moved into plugin space. TM should be improved to be able to hold
sufficient information, and plugins should help by providing that data.
But *just* the data, don't offload very specific use cases onto them,
this will make us less flexible in the long run. If Geany has the data,
we can implement new features inside Geany or non-ft-plugins. Otherwise
we would have to modify every single ft-plugin for new features and
exclude non-ft-plugins.

Post by Lex Trotman

I disagree with the proposed solution for those 4, because they are
offloading logic on a per feature basis to plugins, only because Geany isn't
capable at the moment. If Geany was capable, then there could be 1 solution
for the 4 features and less complexity in each plugin (and we know the
quality of plugins varies a lot so they should have little complexity as
possible).

Encoding the knowledge of language semantics into Geany, for each
language supported, is going to make autocomplete and calltip code
look like c.c. Its not the way to go.

Nobody suggested to encode specific language semantics into Geany. I'm
suggesting TM should become more generic, i.e. TMTag should transport as
much information as required (even if some of it can't be provided by
ctags but only some plugins).

Post by Lex Trotman

The solution I have in mind simply allows plugins to pass tags to Geany
which they parsed with more advanced code. The tags itself would advanced
too, to allow for the improvements current TM+ctags can't offer. Symbol
tree, calltips, autocompletion, jump-to-decl can all be improved based on
the advanced tags.

Well, again you are encoding language semantics into Geany, for
example for C++ that means autocompletion and calltips need to handle
1) local symbol scopes, 2) member functions being in the scope of the
class, even when they are not 3) argument dependent lookup 4) template
expansion lookup and 5) handling of template parameter based typing.
These are hard, just ask the GCC and clang guys. And every user of
Geany will have to pay the cost of the code they don't use, unless
they use C++.
Then for a multidispatch language like Julia you need to handle
overloading in an even more subtle way than C++ overloading.
And why re-implement these language specific subtle and difficult
features in Geany when more and more languages are providing libclang
like libraries to do it for us, accurately and up to date with the
language.

Most of the issues don't apply to just C++ but so many other languages
as well (e.g. Java). All of them can be improved more easily, not just
C++, if Geany can offer a powerful framework for that. But the framework
Matthew proposed is not powerful to me. It just evades the current
problems simply by moving Geany's deficiencies to the plugin space.

Best regards

[Colomban Wendling]

Post by Thomas Martitz

Post by Lex Trotman

[…]

Certainly 1) showing symbols in the symbol list, 2) autocomplete and
3) calltips are currently available to a degree in Geany. But
highlighting, build commands and build result handling are not.
But
to be able to do 2) and 3) accurately needs more knowledge of each
language semantics than is currently available in Geany or tagmanager.

That's right. But it doesn't mean the features should be *entirely*
moved into plugin space. TM should be improved to be able to hold
sufficient information, and plugins should help by providing that data.
But *just* the data, don't offload very specific use cases onto them,
this will make us less flexible in the long run. If Geany has the data,
we can implement new features inside Geany or non-ft-plugins. Otherwise
we would have to modify every single ft-plugin for new features and
exclude non-ft-plugins.

I don't think the real problem for the current state to really only be
about what tags are generated. Sure, if tags for local variables were
generated, scope completion would become more useful just like that
indeed. And then, the current code should only be improved to properly
use the tags to show completions.

But this doesn't take into account several language specific things, like:

* what is an identifier? (for looking up what to complete)
* what is an argument list? (for showing calltips)
* what are the scope semantics?
* …

Sure, some of those can be implemented generically using TagManager
capabilities with appropriate tags, just by improving the generic code.
But that means scope semantics are always the same, or are pluggable,
and same for the other details.

Yes, I'd love TagManager (and our use of it rather) to be as good as it
can get, and it could get pretty far (like what I tried with that
"visible in current scope" patch in the scoped calltip PR).
It's not trivial (like requires to know all the imported namespaces and
stuff), but it's doable if all languages can fit in the same jar [1]

But if a language has totally different semantics than, say, C++, we'd
be pretty much screwed no matter how good the C++ semantics were.

Also, I think Matthew pointed out that in any case, to get perfect
completions we'd need prefect tags, and that we'd have to check how fast
it is with virtually an infinite set of tags (Jiří?). But that's indeed
not a concern to have now.


Anyway, my point is that I totally agree that we should try to get TM
completions to the best they can be, but that no matter how good it is
won't ever be as good (well, at least not better) than completely
specific code. So I think Matthew's goal of allowing to *replace* code
deciding what to complete makes sense.


Regards,
Colomban


[1] I don't know the English expression, so I built one that sounded nice :)

[Lex Trotman]

[...]

Post by Lex Trotman
But
to be able to do 2) and 3) accurately needs more knowledge of each
language semantics than is currently available in Geany or tagmanager.

That's right. But it doesn't mean the features should be *entirely* moved
into plugin space. TM should be improved to be able to hold sufficient
information, and plugins should help by providing that data. But *just* the
data, don't offload very specific use cases onto them, this will make us
less flexible in the long run. If Geany has the data, we can implement new
features inside Geany or non-ft-plugins. Otherwise we would have to modify
every single ft-plugin for new features and exclude non-ft-plugins.

The problem isn't having the data, its what you do with the data. To
repeat two C++ examples I posted on IRC:

namespace foo {
struct A { int a=1; int b=2; };
void f(A& anA, int inc){ anA.a+= inc; }
};

foo::A anA;
std::vector<foo::A> someAs;

The current Geany correctly parses this and shows the correct thing in
the symbols pane, so it has the data, it doesn't even need to use a
plugin for the data.

int i = someAs[1]. // Ok TM show me whatcha got, whats legal here?

Answer: a and b because the vector returns a reference to its template
parameter, here foo::A that has members a and b, you have to expand
the template to find the answer. This is used all over the standard
library.

f( // no calltip because no f is defined in this namespace
f(anA, // so thats the calltip now?

Answer: void foo::f(A& anA, int inc) even though f() is in namespace
foo, because the first argument is type A which is declared in
namespace foo, ADL then will find foo::f(). This is also used in the
standard library and many other libraries.

Both of these are C++ specific semantics, (types generated by template
instantiation and argument dependent lookup). I don't believe TM
should be be expanded to include such knowledge.

[...]

Nobody suggested to encode specific language semantics into Geany. I'm
suggesting TM should become more generic, i.e. TMTag should transport as
much information as required (even if some of it can't be provided by ctags
but only some plugins).

But as noted above you need the language specific knowledge to USE the
data too. A generic TM just can't answer the question. There is no
problem improving TM, but at some point the question is just too
language specific for a geenric TM to handle. So the question becomes
"who knows when that is?" and only the language specific plugin knows
that TM doesn't know what its doing. The plugin has to be asked
first, if it then defers to TM, fine.

Most of the issues don't apply to just C++ but so many other languages as
well (e.g. Java). All of them can be improved more easily, not just C++, if
Geany can offer a powerful framework for that. But the framework Matthew
proposed is not powerful to me. It just evades the current problems simply
by moving Geany's deficiencies to the plugin space.

Because to make Geany able to answer the two questions I asked above
it must have language specific knowledge coded into it (well at least
I know of no other languages with ADL and neither does wikipedia
https://en.wikipedia.org/wiki/Argument-dependent_name_lookup).

Certainly some issues don't belong to C++ alone, no language has local
declarations and scopes handled by TM, and that would help, but again
the specifics of scopes, and lookups is language specific, so you need
language specific code in Geany.

Cheers
Lex

Best regards
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Thomas Martitz]

Post by Lex Trotman
[...]

Post by Lex Trotman
But
to be able to do 2) and 3) accurately needs more knowledge of each
language semantics than is currently available in Geany or tagmanager.

That's right. But it doesn't mean the features should be *entirely* moved
into plugin space. TM should be improved to be able to hold sufficient
information, and plugins should help by providing that data. But *just* the
data, don't offload very specific use cases onto them, this will make us
less flexible in the long run. If Geany has the data, we can implement new
features inside Geany or non-ft-plugins. Otherwise we would have to modify
every single ft-plugin for new features and exclude non-ft-plugins.

The problem isn't having the data, its what you do with the data. To
namespace foo {
struct A { int a=1; int b=2; };
void f(A& anA, int inc){ anA.a+= inc; }
};
foo::A anA;
std::vector<foo::A> someAs;
The current Geany correctly parses this and shows the correct thing in
the symbols pane, so it has the data, it doesn't even need to use a
plugin for the data.
int i = someAs[1]. // Ok TM show me whatcha got, whats legal here?
Answer: a and b because the vector returns a reference to its template
parameter, here foo::A that has members a and b, you have to expand
the template to find the answer. This is used all over the standard
library.
f( // no calltip because no f is defined in this namespace
f(anA, // so thats the calltip now?
Answer: void foo::f(A& anA, int inc) even though f() is in namespace
foo, because the first argument is type A which is declared in
namespace foo, ADL then will find foo::f(). This is also used in the
standard library and many other libraries.
Both of these are C++ specific semantics, (types generated by template
instantiation and argument dependent lookup). I don't believe TM
should be be expanded to include such knowledge.

Why not? TM could have separate tags for each known template instance
and know which of these applies to someAs, based on information provided
by a ft-plugin. Then the rest would work.

Besides, template instantiation requires compiling the units IIUC, which
is probably not gonna happen ever? At least not on every keystroke as
done currently.

Best regards.

[Lex Trotman]

Why not? TM could have separate tags for each known template instance and
know which of these applies to someAs, based on information provided by a
ft-plugin. Then the rest would work.

Yes, if Geany can be made to understand that `vector<int>` and
`vector< int >` and `vector <int >` are all the same type so it will
look them up correctly. In parameterised generics, what used to be a
type name is now a type syntax, and no matter how I type it, its the
same.

As an example Geany trying to lookup something like
`template_name<params>::member` to get the type of `member` has to
apply syntax analysis to the `template_name<params>` part and can no
longer just get a name.

And parameterised generics are available in a number of languages, but
with differing syntax (`name{params}` is a popular one for non-{}
languages) and again with differing semantics. So thats even more
language specific knowledge to be encoded in Geany, either in TM or in
Geany if TM doesn't provide the facility.

Much easier to ask the language specific plugin what
`template_name<params>::member` or `type_name{params}` is.

Besides, template instantiation requires compiling the units IIUC, which is
probably not gonna happen ever? At least not on every keystroke as done
currently.

Well, analysing them yes, and it will probably have to happen in a
separate thread/process, so again it isn't something thats going to be
built-in to Geany.

Best regards.
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Matthew Brush]

Post by Thomas Martitz

Post by Lex Trotman
[...]
Both of these are C++ specific semantics, (types generated by template
instantiation and argument dependent lookup). I don't believe TM
should be be expanded to include such knowledge.

Why not? TM could have separate tags for each known template instance
and know which of these applies to someAs, based on information provided
by a ft-plugin. Then the rest would work.
Besides, template instantiation requires compiling the units IIUC, which
is probably not gonna happen ever? At least not on every keystroke as
done currently.

I can't speak to all compiler libraries, but at least libclang,
libpython and libvala "compile" the source (well just the front-end of
the compiler is needed). They literally use the built-in compiler front
ends to understand the code. In the case of libclang, it additionally
provides helpful methods for performing IDE-related features on the AST,
while say with libvala, the ft-plugin would be required to perform it's
own analysis of the AST to implement those feaures, which is still a lot
less work than in Geany/TM since it has access to the full AST/context
and the ft-plugin need not fear encoding language specific semantics
into its logic.

Cheers,
Matthew Brush

[Thomas Martitz]

Post by Matthew Brush
I can't speak to all compiler libraries, but at least libclang,
libpython and libvala "compile" the source (well just the front-end of
the compiler is needed). They literally use the built-in compiler
front ends to understand the code. In the case of libclang, it
additionally provides helpful methods for performing IDE-related
features on the AST, while say with libvala, the ft-plugin would be
required to perform it's own analysis of the AST to implement those
feaures, which is still a lot less work than in Geany/TM since it has
access to the full AST/context and the ft-plugin need not fear
encoding language specific semantics into its logic.

How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them
from?

Best regards

[Lex Trotman]

Post by Thomas Martitz

I can't speak to all compiler libraries, but at least libclang, libpython
and libvala "compile" the source (well just the front-end of the compiler is
needed). They literally use the built-in compiler front ends to understand
the code. In the case of libclang, it additionally provides helpful methods
for performing IDE-related features on the AST, while say with libvala, the
ft-plugin would be required to perform it's own analysis of the AST to
implement those feaures, which is still a lot less work than in Geany/TM
since it has access to the full AST/context and the ft-plugin need not fear
encoding language specific semantics into its logic.

How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them
from?

Libclang needs the full-fat build knowledge that "project" systems on
other IDEs provide. Another reason to keep it separate from Geany.

Post by Thomas Martitz
Best regards
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Thomas Martitz]

Post by Lex Trotman

Post by Thomas Martitz

I can't speak to all compiler libraries, but at least libclang, libpython
and libvala "compile" the source (well just the front-end of the compiler is
needed). They literally use the built-in compiler front ends to understand
the code. In the case of libclang, it additionally provides helpful methods
for performing IDE-related features on the AST, while say with libvala, the
ft-plugin would be required to perform it's own analysis of the AST to
implement those feaures, which is still a lot less work than in Geany/TM
since it has access to the full AST/context and the ft-plugin need not fear
encoding language specific semantics into its logic.

How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them
from?

Libclang needs the full-fat build knowledge that "project" systems on
other IDEs provide. Another reason to keep it separate from Geany.

So one would have to adjust the build settings and one or more
ft-plugins all the time?

IMO, complex build system integration is out of scope for ft-plugins.
But I see that not all features can be supported properly without build
system.

Best regards

[Lex Trotman]

Post by Lex Trotman

Post by Thomas Martitz

I can't speak to all compiler libraries, but at least libclang, libpython
and libvala "compile" the source (well just the front-end of the compiler is
needed). They literally use the built-in compiler front ends to understand
the code. In the case of libclang, it additionally provides helpful methods
for performing IDE-related features on the AST, while say with libvala, the
ft-plugin would be required to perform it's own analysis of the AST to
implement those feaures, which is still a lot less work than in Geany/TM
since it has access to the full AST/context and the ft-plugin need not fear
encoding language specific semantics into its logic.

How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them
from?

Libclang needs the full-fat build knowledge that "project" systems on
other IDEs provide. Another reason to keep it separate from Geany.

So one would have to adjust the build settings and one or more ft-plugins
all the time?

I am guessing that one of the reasons Matthew mentioned build settings
access in #1195 is so that the plugins with complex build requirements
can set Geany's simple build system to match their complex one.

IMO, complex build system integration is out of scope for ft-plugins. But I
see that not all features can be supported properly without build system.

I am not sure I understand you? That sounds like you are saying that
plugins cannot be allowed to have complex build systems inside the
plugin for its own use, or do I misunderstand?

Best regards
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Matthew Brush]

Post by Thomas Martitz

Post by Lex Trotman

Post by Thomas Martitz

Post by Matthew Brush
I can't speak to all compiler libraries, but at least
libclang, libpython and libvala "compile" the source (well just
the front-end of the compiler is needed). They literally use
the built-in compiler front ends to understand the code. In the
case of libclang, it additionally provides helpful methods for
performing IDE-related features on the AST, while say with
libvala, the ft-plugin would be required to perform it's own
analysis of the AST to implement those feaures, which is still
a lot less work than in Geany/TM since it has access to the
full AST/context and the ft-plugin need not fear encoding
language specific semantics into its logic.

How do you pass {C,CXX,CPP}FLAGS to libclang? And where do you get them
from?

Libclang needs the full-fat build knowledge that "project" systems on
other IDEs provide. Another reason to keep it separate from Geany.

So one would have to adjust the build settings and one or more
ft-plugins all the time?

One would have to tell the ft-plugin how to compile the code (where
applicable). libclang can read a so-called "compilation-database"[0]
which can be generated by CMake[1], Ninja[2], a Make wrapper called
Bear[3], or written by hand.

Post by Thomas Martitz
IMO, complex build system integration is out of scope for ft-plugins.

I generally agree, although we could potentially look at improving
projects/build commands in the future to some extent, I don't think we
should look at it much at this point with ft-plugins.

Post by Thomas Martitz
But I see that not all features can be supported properly without build
system.

I don't think they'll usually require a "build system" per se, but they
definitively need to be told how to compile the code where applicable.

For libpython, I don't think it needs any flags. For libvala IIRC and
libclang (if not using compile_commands.json) they require the `argv`
you'd pass to the compiler itself. It may very well be possible for
ft-plugins using libraries with the `argv` approach to just grab the
compile command from Geany's build commands and convert it back into an
`argv` array or something.

Cheers,
Matthew Brush


[0]: http://clang.llvm.org/docs/JSONCompilationDatabase.html
[1]:
https://cmake.org/cmake/help/v3.5/variable/CMAKE_EXPORT_COMPILE_COMMANDS.html
[2]: https://ninja-build.org/manual.html#_extra_tools
[3]: https://github.com/rizsotto/Bear

[Thomas Martitz]

Post by Matthew Brush
I don't think they'll usually require a "build system" per se, but
they definitively need to be told how to compile the code where
applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and
libclang (if not using compile_commands.json) they require the `argv`
you'd pass to the compiler itself. It may very well be possible for
ft-plugins using libraries with the `argv` approach to just grab the
compile command from Geany's build commands and convert it back into
an `argv` array or something.

Not sure how you use Geany, but I never set the cc command line for
individual files. I never type CFLAGS or CXXFLAGS into Geany's build
settings.

Actually most of the time I don't set up the build system at all and
alt-tab to a terminal where I use my employers crazy build system which
uses make under the hood but can't be used from within Geany properly.

Then, some other time (e.g. for compiling Geany) I sometimes use the
make command. But obviously Geany only knows that running "make" builds
something. It doesn't have enough information to construct a compiler
command line.

Some other time again I'm working on cross compiled projects. Here I
again don't usually use Geany's build system support, but if I would it
would run (e.g.) avr-gcc, with avr-gcc specific compiler flags, not
known to clang (is there even a clang port for avr? I don't think so).

So a ft-plugin that only works if a file can be compiled is likely to be
unworkable for me. It needs to deal with when it can't build the source
files (either because of lacking *FLAGS or because the target isn't
supported by the compiler). Likewise, if a ft-plugin only supports x86
(i.e. is arch dependent) it's not interesting to me.

There is also the possibility that the file I'm editing is not compiled
at all with the current build settings, e.g. win32.c.

How would you deal with all that? The ft-plugin has to do something in
such situations doesnt it?

Best regards.

[Matthew Brush]

Post by Thomas Martitz

Post by Matthew Brush
I don't think they'll usually require a "build system" per se, but
they definitively need to be told how to compile the code where
applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and
libclang (if not using compile_commands.json) they require the `argv`
you'd pass to the compiler itself. It may very well be possible for
ft-plugins using libraries with the `argv` approach to just grab the
compile command from Geany's build commands and convert it back into
an `argv` array or something.

Not sure how you use Geany, but I never set the cc command line for
individual files. I never type CFLAGS or CXXFLAGS into Geany's build
settings.

Yeah, that's why the absolute base functionality provided must always
exist. I'd probably be more willing to do a little setup for actual
projects if there was a benefit though.

Post by Thomas Martitz
Actually most of the time I don't set up the build system at all and
alt-tab to a terminal where I use my employers crazy build system which
uses make under the hood but can't be used from within Geany properly.

This use-case should continue to be the default supported one.

Post by Thomas Martitz
Then, some other time (e.g. for compiling Geany) I sometimes use the
make command. But obviously Geany only knows that running "make" builds
something. It doesn't have enough information to construct a compiler
command line.

Indeed. To provide "real IDE" features and project requires more than
what Geany provides out of the box. I haven't really proposed a design
for this, so as it stands ft-plugins would have to provide some means
(ex. advanced project support, compile_commands.json, custom GUI, etc)
to get this info from the user.

Post by Thomas Martitz
Some other time again I'm working on cross compiled projects. Here I
again don't usually use Geany's build system support, but if I would it
would run (e.g.) avr-gcc, with avr-gcc specific compiler flags, not
known to clang (is there even a clang port for avr? I don't think so).

For an example libclang plugin, clang would have to be able to parse the
source. It would require valid C/C++/Obj-C code for sure (though IIRC
some of the helper functions for IDEs/tools handle partial source files
or simple lexing to handle on-the-fly IDE highlighting and such).

Post by Thomas Martitz
So a ft-plugin that only works if a file can be compiled is likely to be
unworkable for me. It needs to deal with when it can't build the source
files (either because of lacking *FLAGS or because the target isn't
supported by the compiler). Likewise, if a ft-plugin only supports x86
(i.e. is arch dependent) it's not interesting to me.

For actual projects you would work on for a while, it would be worth
setting up some meta-stuff.

Post by Thomas Martitz
There is also the possibility that the file I'm editing is not compiled
at all with the current build settings, e.g. win32.c.
How would you deal with all that? The ft-plugin has to do something in
such situations doesnt it?

Fallback to the current features, ideally.

Cheers,
Matthew Brush

[Thomas Martitz]

Post by Matthew Brush

Post by Thomas Martitz

Post by Matthew Brush
I don't think they'll usually require a "build system" per se, but
they definitively need to be told how to compile the code where
applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and
libclang (if not using compile_commands.json) they require the `argv`
you'd pass to the compiler itself. It may very well be possible for
ft-plugins using libraries with the `argv` approach to just grab the
compile command from Geany's build commands and convert it back into
an `argv` array or something.

Not sure how you use Geany, but I never set the cc command line for
individual files. I never type CFLAGS or CXXFLAGS into Geany's build
settings.

Yeah, that's why the absolute base functionality provided must always
exist. I'd probably be more willing to do a little setup for actual
projects if there was a benefit though.

Post by Thomas Martitz
Actually most of the time I don't set up the build system at all and
alt-tab to a terminal where I use my employers crazy build system which
uses make under the hood but can't be used from within Geany properly.

This use-case should continue to be the default supported one.

But this can't be supported by a libclang ft-plugin, can it? Are you
suggesting the default use-case bypasses the ft-plugin? That seems weird
to begin with.

Post by Matthew Brush

Post by Thomas Martitz
Then, some other time (e.g. for compiling Geany) I sometimes use the
make command. But obviously Geany only knows that running "make" builds
something. It doesn't have enough information to construct a compiler
command line.

Indeed. To provide "real IDE" features and project requires more than
what Geany provides out of the box. I haven't really proposed a design
for this, so as it stands ft-plugins would have to provide some means
(ex. advanced project support, compile_commands.json, custom GUI, etc)
to get this info from the user.

A solution for this should be part of the discussion, otherwise we're
going end up with a dozen completely ways and UIs configure ft-plugins.
Which is going to be a PITA for users if the ft-plugins have to be
regularly reconfigured on a per-project basis.

Post by Matthew Brush

Post by Thomas Martitz
Some other time again I'm working on cross compiled projects. Here I
again don't usually use Geany's build system support, but if I would it
would run (e.g.) avr-gcc, with avr-gcc specific compiler flags, not
known to clang (is there even a clang port for avr? I don't think so).

For an example libclang plugin, clang would have to be able to parse
the source. It would require valid C/C++/Obj-C code for sure (though
IIRC some of the helper functions for IDEs/tools handle partial source
files or simple lexing to handle on-the-fly IDE highlighting and such).

Post by Thomas Martitz
So a ft-plugin that only works if a file can be compiled is likely to be
unworkable for me. It needs to deal with when it can't build the source
files (either because of lacking *FLAGS or because the target isn't
supported by the compiler). Likewise, if a ft-plugin only supports x86
(i.e. is arch dependent) it's not interesting to me.

For actual projects you would work on for a while, it would be worth
setting up some meta-stuff.

I'm afraid I won't be able. The CFLAGS depend on the source file I edit.
There are multiple projects involved (e.g. Linux kernel and user space),
there is not just the One Project.

The more I'm thinking about it...if key functionality depends on being
able to compile the file, I won't be able to use a libclang based ft-plugin.

Really, this is one reason why I use Geany in the first place. Other
IDEs such as eclipse require extensive, per-project build setup, to give
even basic symbol parsing support. Such IDEs are generally limited to
select work flows, such as Makefile-based, which are often incompatible
with mine e.g. the work flow at my workplace. I hugely appreciate that
Geany gives actually (IMO) decent auto-completion and calltips out of
the box (and can be improved with just setting a few file patterns with
ProjectOrganizer), for any file I throw at it.

I do see that this is valuable for other people and languages so I'm not
fundamentally opposed to do actual compiling (if done right). But there
needs to be a decent fallback when this is not possible, even if it just
means to fallback to Geany's builtin support (so I'd likely always fall
back).

Best regards

[Matthew Brush]

Post by Thomas Martitz

Post by Matthew Brush

Post by Thomas Martitz

Post by Matthew Brush
I don't think they'll usually require a "build system" per se, but
they definitively need to be told how to compile the code where
applicable.
For libpython, I don't think it needs any flags. For libvala IIRC and
libclang (if not using compile_commands.json) they require the `argv`
you'd pass to the compiler itself. It may very well be possible for
ft-plugins using libraries with the `argv` approach to just grab the
compile command from Geany's build commands and convert it back into
an `argv` array or something.

Not sure how you use Geany, but I never set the cc command line for
individual files. I never type CFLAGS or CXXFLAGS into Geany's build
settings.

Yeah, that's why the absolute base functionality provided must always
exist. I'd probably be more willing to do a little setup for actual
projects if there was a benefit though.

Post by Thomas Martitz
Actually most of the time I don't set up the build system at all and
alt-tab to a terminal where I use my employers crazy build system which
uses make under the hood but can't be used from within Geany properly.

This use-case should continue to be the default supported one.

But this can't be supported by a libclang ft-plugin, can it? Are you
suggesting the default use-case bypasses the ft-plugin? That seems weird
to begin with.

It could fall-back to the default functionality if no ft-plugin performs
a given feature.

Post by Thomas Martitz

Post by Matthew Brush

Post by Thomas Martitz
Then, some other time (e.g. for compiling Geany) I sometimes use the
make command. But obviously Geany only knows that running "make" builds
something. It doesn't have enough information to construct a compiler
command line.

Indeed. To provide "real IDE" features and project requires more than
what Geany provides out of the box. I haven't really proposed a design
for this, so as it stands ft-plugins would have to provide some means
(ex. advanced project support, compile_commands.json, custom GUI, etc)
to get this info from the user.

A solution for this should be part of the discussion, otherwise we're
going end up with a dozen completely ways and UIs configure ft-plugins.
Which is going to be a PITA for users if the ft-plugins have to be
regularly reconfigured on a per-project basis.

It's going to be plugin/language specific anyways. I agree it should be
considered eventually, but it's not required to provide basic hooks for
ft-plugins to start providing some features.

Post by Thomas Martitz

Post by Matthew Brush

Post by Thomas Martitz
Some other time again I'm working on cross compiled projects. Here I
again don't usually use Geany's build system support, but if I would it
would run (e.g.) avr-gcc, with avr-gcc specific compiler flags, not
known to clang (is there even a clang port for avr? I don't think so).

For an example libclang plugin, clang would have to be able to parse
the source. It would require valid C/C++/Obj-C code for sure (though
IIRC some of the helper functions for IDEs/tools handle partial source
files or simple lexing to handle on-the-fly IDE highlighting and such).

Post by Thomas Martitz
So a ft-plugin that only works if a file can be compiled is likely to be
unworkable for me. It needs to deal with when it can't build the source
files (either because of lacking *FLAGS or because the target isn't
supported by the compiler). Likewise, if a ft-plugin only supports x86
(i.e. is arch dependent) it's not interesting to me.

If your code can't be (cross) compiled, I'm afraid you're screwed anyway :)

For some stuff, like partial code (ex. you're still typing it) or
guarded out by platform macros, I'd expect some some reduced
functionality, or different behaviour, at least.

To test it out, just try a competent full-fledge IDE such as QtCreator,
Visual Studio, XCode, or Eclipse (among many others) and see what it does.

Post by Thomas Martitz

Post by Matthew Brush
For actual projects you would work on for a while, it would be worth
setting up some meta-stuff.

I'm afraid I won't be able. The CFLAGS depend on the source file I edit.
There are multiple projects involved (e.g. Linux kernel and user space),
there is not just the One Project.

If you can't describe your build, yeah it's not so useful.

Post by Thomas Martitz
The more I'm thinking about it...if key functionality depends on being
able to compile the file, I won't be able to use a libclang based ft-plugin.

In order to provide "smart" features requires to understand the code. If
you can't describe your build and make a regular c/c++/obj-c (cross)
compiler able to handle your code, I'm afraid at least my CDK/libclang
ft-plugin indeed wouldn't be of use to you. Doesn't mean others used to
normal IDEs with describable builds won't benefit though.

Post by Thomas Martitz
Really, this is one reason why I use Geany in the first place. Other
IDEs such as eclipse require extensive, per-project build setup, to give
even basic symbol parsing support. Such IDEs are generally limited to
select work flows, such as Makefile-based, which are often incompatible
with mine e.g. the work flow at my workplace. I hugely appreciate that
Geany gives actually (IMO) decent auto-completion and calltips out of
the box (and can be improved with just setting a few file patterns with
ProjectOrganizer), for any file I throw at it.

There is no desire to remove this base functionality, only to enhance it
for people who want more than you from Geany without bloating the core.

Post by Thomas Martitz
I do see that this is valuable for other people and languages so I'm not
fundamentally opposed to do actual compiling (if done right). But there
needs to be a decent fallback when this is not possible, even if it just
means to fallback to Geany's builtin support (so I'd likely always fall
back).

That would be ideal.

Cheers,
Matthew Brush

[Colomban Wendling]

Post by Matthew Brush
[…]
Syntax Highlighting
-------------------
Most likely using an API based on/similar to Scintilla's "container lexers".
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*,
guint start_pos, guint end_pos, gpointer user_data);
As with Scintilla's "container lexer", it would just tell the provider
what and where to highlight. It might be pointless providing `end_pos`
it could probably just highlight a whole line at time (maybe like
Scintilla's 'style-needed' notification).

I'm really not sure it's a good idea to go the custom callback way.
IMO, we should first try and see how easy it'd be with plugins providing
their own full-blown Scintilla lexer library that we just add and use.

Having our own callback means one more indirection, and changing the
SciLexer to CONTAINER anyway, so I don't see much advantage just now.

[Matthew Brush]

Post by Colomban Wendling

Post by Matthew Brush
[…]
Syntax Highlighting
-------------------
Most likely using an API based on/similar to Scintilla's "container lexers".
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*,
guint start_pos, guint end_pos, gpointer user_data);
As with Scintilla's "container lexer", it would just tell the provider
what and where to highlight. It might be pointless providing `end_pos`
it could probably just highlight a whole line at time (maybe like
Scintilla's 'style-needed' notification).

I'm really not sure it's a good idea to go the custom callback way.
IMO, we should first try and see how easy it'd be with plugins providing
their own full-blown Scintilla lexer library that we just add and use.

The only positive I really see, which in practice probably won't exist,
is modularity and ability to re-use lexers independent of
Geany/ft-plugin (ie. for all Scintilla-using apps). I say in practice
because at least with my `LexClang.so` I needed it to be bound into
Geany anyway to get hooks for when to re-parse (you can't re-parse a
million token C++ file each time Scintilla wants to re-colour a line of
code). Further, the dynamic lexer needs to cooperate with
Geany/ft-plugin, or at least deviate from normal Scintilla lexers, if it
wanted to provide/setup its own lexical states/styles (TBD how this part
will go).

Post by Colomban Wendling
Having our own callback means one more indirection, and changing the
SciLexer to CONTAINER anyway, so I don't see much advantage just now.

With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not
to fit well (too isolated, too many assumptions that it's a simple dumb
lexer and not a semantic-based on, etc) . All I really wanted was a way
to disable Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`)
without changing the filetype in Geany, and without doing it behind
Geany's back from the plugin.

Cheers,
Matthew Brush

[Lex Trotman]

All I really wanted was a way to disable

Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the
filetype in Geany, and without doing it behind Geany's back from the plugin.

In fact some of the tools using scintilla and which provide a richer
styling do exactly that, probably for all the reasons Matthew notes.

Cheers,
Matthew Brush
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Lex Trotman]

Post by Colomban Wendling

Post by Matthew Brush
[…]
Syntax Highlighting
-------------------
Most likely using an API based on/similar to Scintilla's "container lexers".
gboolean (*highlight)(GeanyPlugin*, GeanyDocument*,
guint start_pos, guint end_pos, gpointer user_data);
As with Scintilla's "container lexer", it would just tell the provider
what and where to highlight. It might be pointless providing `end_pos`
it could probably just highlight a whole line at time (maybe like
Scintilla's 'style-needed' notification).

I'm really not sure it's a good idea to go the custom callback way.
IMO, we should first try and see how easy it'd be with plugins providing
their own full-blown Scintilla lexer library that we just add and use.

The only positive I really see, which in practice probably won't exist, is
modularity and ability to re-use lexers independent of Geany/ft-plugin (ie.
for all Scintilla-using apps). I say in practice because at least with my
`LexClang.so` I needed it to be bound into Geany anyway to get hooks for
when to re-parse (you can't re-parse a million token C++ file each time
Scintilla wants to re-colour a line of code). Further, the dynamic lexer
needs to cooperate with Geany/ft-plugin, or at least deviate from normal
Scintilla lexers, if it wanted to provide/setup its own lexical
states/styles (TBD how this part will go).

Post by Colomban Wendling
Having our own callback means one more indirection, and changing the
SciLexer to CONTAINER anyway, so I don't see much advantage just now.

With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to
fit well (too isolated, too many assumptions that it's a simple dumb lexer
and not a semantic-based on, etc) . All I really wanted was a way to disable
Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the
filetype in Geany, and without doing it behind Geany's back from the plugin.

Agree with Matthew.

The point of not using the standard Scintilla lexer is to be able to
add semantic information made available for the language support
library in the FT-Plugin.

The major example of this is fixing the current situation where any
name that is a type in any scope is coloured as a type in every other
scope, even if the type is not visible there.

That means that the lexer is intimately tied into the FT-Plugin so it
likely won't run without the FT-Plugin anyway, so actually including
it in the plugin and accessing it via the container lexers interface
looks better than having a separate DLL that won't run by itself
anyway and then has to be sure its loaded at the right time.

Cheers
Lex

Cheers,
Matthew Brush
_______________________________________________
Devel mailing list
https://lists.geany.org/cgi-bin/mailman/listinfo/devel

[Thomas Martitz]

Post by Lex Trotman

With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to
fit well (too isolated, too many assumptions that it's a simple dumb lexer
and not a semantic-based on, etc) . All I really wanted was a way to disable
Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the
filetype in Geany, and without doing it behind Geany's back from the plugin.

Agree with Matthew.
The point of not using the standard Scintilla lexer is to be able to
add semantic information made available for the language support
library in the FT-Plugin.
The major example of this is fixing the current situation where any
name that is a type in any scope is coloured as a type in every other
scope, even if the type is not visible there.
That means that the lexer is intimately tied into the FT-Plugin so it
likely won't run without the FT-Plugin anyway, so actually including
it in the plugin and accessing it via the container lexers interface
looks better than having a separate DLL that won't run by itself
anyway and then has to be sure its loaded at the right time.

Can this support color schemes and custom keywords (although the latter
is probably not very important).

I don't know what SCLEX_CONTAINER is.

Best regards

[Matthew Brush]

Post by Thomas Martitz

Post by Lex Trotman

Post by Matthew Brush
With the `LexClang.so` dynamic lexer I made, dynamic lexers
seemed not to fit well (too isolated, too many assumptions that
it's a simple dumb lexer and not a semantic-based on, etc) . All
I really wanted was a way to disable Scintilla's lexer (ie.
switch it to `SCLEX_CONTAINER`) without changing the filetype in
Geany, and without doing it behind Geany's back from the plugin.

Agree with Matthew.
The point of not using the standard Scintilla lexer is to be able to
add semantic information made available for the language support
library in the FT-Plugin.
The major example of this is fixing the current situation where any
name that is a type in any scope is coloured as a type in every other
scope, even if the type is not visible there.
That means that the lexer is intimately tied into the FT-Plugin so it
likely won't run without the FT-Plugin anyway, so actually including
it in the plugin and accessing it via the container lexers interface
looks better than having a separate DLL that won't run by itself
anyway and then has to be sure its loaded at the right time.

Can this support color schemes and custom keywords (although the latter
is probably not very important).
I don't know what SCLEX_CONTAINER is.

It's the built-in (application-provided) lexing support in Scintilla. It
means "Tell me when, and I'll highlight the source for you". It only
requires a GTK+ signal from Scintilla rather than having to implement a
concrete C++ class of ILexer and conforming to some interface in a
separate DLL, to perform roughly the same task.

http://www.scintilla.org/ScintillaDoc.html#SCN_STYLENEEDED

Cheers,
Matthew Brush

[Matthew Brush]

Post by Lex Trotman

Post by Colomban Wendling
[...]
Having our own callback means one more indirection, and changing the
SciLexer to CONTAINER anyway, so I don't see much advantage just now.

With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not to
fit well (too isolated, too many assumptions that it's a simple dumb lexer
and not a semantic-based on, etc) . All I really wanted was a way to disable
Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`) without changing the
filetype in Geany, and without doing it behind Geany's back from the plugin.

Agree with Matthew.
The point of not using the standard Scintilla lexer [...]

I wouldn't even call it "standard" lexer. Though it conforms to the same
C++ interface as a few other modern, well-maintained lexers, AFAIK only
handful of Scintilla's lexers (ex. LexCpp) are actually able to compile
as dynamic libraries.

One of the few useful dynamic lexers I've ever seen is Scintillua[0] and
it actually makes some kind of sense to be a generic dynamic lexer here
since it can proxy for other lexers using a completely different
non-Scintilla mechanisms internally (ie. Lua and PEGs).

Cheers,
Matthew Brush

[0]: http://foicica.com/scintillua/

[Colomban Wendling]

Post by Matthew Brush

Post by Colomban Wendling

[…]

I'm really not sure it's a good idea to go the custom callback way.
IMO, we should first try and see how easy it'd be with plugins providing
their own full-blown Scintilla lexer library that we just add and use.

The only positive I really see, which in practice probably won't exist,
is modularity and ability to re-use lexers independent of
Geany/ft-plugin (ie. for all Scintilla-using apps). I say in practice
because at least with my `LexClang.so` I needed it to be bound into
Geany anyway to get hooks for when to re-parse (you can't re-parse a
million token C++ file each time Scintilla wants to re-colour a line of
code). Further, the dynamic lexer needs to cooperate with
Geany/ft-plugin, or at least deviate from normal Scintilla lexers, if it
wanted to provide/setup its own lexical states/styles (TBD how this part
will go).

Post by Colomban Wendling
Having our own callback means one more indirection, and changing the
SciLexer to CONTAINER anyway, so I don't see much advantage just now.

With the `LexClang.so` dynamic lexer I made, dynamic lexers seemed not
to fit well (too isolated, too many assumptions that it's a simple dumb
lexer and not a semantic-based on, etc) . All I really wanted was a way
to disable Scintilla's lexer (ie. switch it to `SCLEX_CONTAINER`)
without changing the filetype in Geany, and without doing it behind
Geany's back from the plugin.

Okay then, if there's good reasons to do so, fine.

I just thought that it would be handier if later on plugins want to be
able to add new filetypes altogether (e.g. not necessarily based on
lib<language>, but just filetypes), but maybe not, or maybe it's not
good for all cases.

Regards,
Colomban