Sign in to follow this  
Followers 0
monoceres

Adding a new lexer to scite/scintilla

21 posts in this topic

I'm currently writing a programming language as a part of a school project.

I'm almost done and now I need to wrap it up in a nice package.

SciTe is a great editor, and more importantly, quite easy to modify with existing lexers.

My language is quite similar to autoit so at first I just tweaked the au3.properties file in scite.

However there are many differences between my language and autoit, one example is comments which are of C/C++ style. This obviously doesn't work with the autoit lexer.

So now I want to write the lexer by myself. I've downloaded the scintilla source code and tried to build it. No problems there, I even tried changing stuff in the autoit lexer to see that everything worked.

Now comes the problems. Where do I start!? After going through the package I found python scripts updating the C code (oh that's never a good sign, code that updates code), enumerations and non-specific readme's that suffers from explaining detailed stuff very good but completely misses the basic "get started" points.

So what's the specific steps needed to get a extremely simple parser working? If I can get a custom lexer that only makes "// this is a comment" green I'm sure I could manage from there.

To my understanding this properties file is all that's needed to get the lexer working in SciTe:

# Skua is the name of my language
file.patterns.skua=*.skua;*.skua


filter.skua=Skua (skua)|$(file.patterns.skua)|

lexer.$(file.patterns.skua)=skua


# Comment
#style.skua.1=fore:#007F00,$(font.comment)

Thanks!


Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites



No idea... But will we be seeing the finished language :) ??

Sure, it has some interesting features such as separate compilation, bytecode compilation and optimized recursion. But it depends on both ruby and java so I doubt many people will try it :(


Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

Have you seen this page that describes the steps? : http://www.scintilla.org/SciTELexer.html

I have no idea how I managed to miss that page. It looks just what I want.

I'm sure I'll be back with more questions :(

Thanks!


Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

Why not create a external scite lexer? (SciTEExternalLexer)

Would love to be able to create on myself for a particular language I'm playing around with. but ... o well.

That was my first thought. But it's just as much work and I have no problems building the entire package. Sticking with the original project is much easier.


Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

Ok, I'm really confused. As soon as I call styler.ColorTo() I crash scite.

What am I doing wrong here?

static void ColouriseSkuaDoc(unsigned int startPos, int length, int initStyle,WordList *keywordlists[],Accessor &styler) {


    int lengthDoc = startPos + length ;

    styler.ColourTo(lengthDoc - 1, SCE_SKUA_COMMENT);

}

Properties file looks like:

# Skua is the name of my language
file.patterns.skua=*.skua;*.skua


filter.skua=Skua (skua)|$(file.patterns.skua)|

lexer.$(file.patterns.skua)=skua

keywords.$(file.patterns.skua)=Func EndFunc

# Comment
style.skua.0=fore:#000000,$(font.comment)
style.skua.1=fore:#007F00,$(font.comment)

Also,

is it okay to take the easy way out and build the lexer with regular expressions? :(

Edited by monoceres

Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

I managed to clear it out. Adding:

StyleContext sc = StyleContext(startPos,startPos+length,SCE_SKUA_DEFAULT,styler);

Fixed it. In some way. I have no idea.


Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

Before I dive in any further - is it possible to add auto-intend without touching the lexer? I've seen the intend-auto directive but I have not been able to get it to work.


Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

Do you perhaps mean indent?

Share this post


Link to post
Share on other sites

Indentation is not controlled by the lexer, it's controlled by a series of properties. It can also be controlled by Lua (some of AutoIt's indentation is handled via Lua).

Share this post


Link to post
Share on other sites

Heh, I never seem to spell indentation right.

I don't get it though. What's wrong with my properties file?

file.patterns.skua=*.skua;

filter.skua=Skua (skua)|$(file.patterns.skua)|

lexer.$(file.patterns.skua)=skua

keywords.$(file.patterns.skua)=FUNC ENDFUNC FOR IN ENDFOR IF ELSEIF ELSE ENDIF WHILE ENDWHILE LOCAL RETURN BREAK CONTINUE
keywords2.$(file.patterns.skua)=PRINT INPUT TOSTRING TYPEOF STRINGLENGTH ABSOLUTE STARTTIMER CALCTIMER RANGE SLEEP \
                                SQRT ALERT ARRAYLENGTH ARRAY CALL RANDOM 



statement.indent.$(file.patterns.skua)=5 Func EndFunc


tabsize=2
tab.size.$(file.patterns.skua)=2
indent.size=2
indent.size.$(file.patterns.skua)=2
use.tabs=1
use.tabs.$(file.patterns.skua)=1
indent.auto=1
tab.indents=2
backspace.unindents=2
indent.automatic=1
indent.opening=Func
indent.closing=EndFunc
indent.maintain.$(file.patterns.skua)=1
    

style.skua.0=fore:#000000
style.skua.1=fore:#007F00
style.skua.2=fore:#00007f,bold
style.skua.3=fore:#ee0000
style.skua.4=fore:#CC33FF
style.skua.5=fore:#006633,bold
style.skua.8=fore:#0080B5

command.go.*.skua=rubyw "$(SciteDefaultHome)\..\skuac.rb" "$(FileNameExt)"
command.build.*.skua=java -jar "$(SciteDefaultHome)\..\skuavm.jar" "$(FileName).sbc"

Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

#14 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

I don't get it though. What's wrong with my properties file?

In relation to ... what problem?

Getting auto-indentation to work. I wish scite to increase the indentation two tabs after a line that contains "Func".


Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

You really need to read the SciTE documentation. The block.start and block.end properties are what control block indentation. Look at the AutoIt properties file to see how it is set up. Also... read the SciTE documentation about the properties.

Share this post


Link to post
Share on other sites

#17 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

MvGulik, SciTE 2.10 has support for writing lexers in Lua. Any other (official) version of SciTE offering that functionality is out-of-date.

Share this post


Link to post
Share on other sites

#19 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

It's a bit easier to add functionality when you actually read the docs, instead of cutting pieces form other property files :(

Only auto-complete left, then it's magical!


Broken link? PM me and I'll send you the file!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0