Adding a new lexer to scite/scintilla

[monoceres]

I'm currently writing a programming language as a part of a school project.

I'm almost done and now I need to wrap it up in a nice package.

SciTe is a great editor, and more importantly, quite easy to modify with existing lexers.

My language is quite similar to autoit so at first I just tweaked the au3.properties file in scite.

However there are many differences between my language and autoit, one example is comments which are of C/C++ style. This obviously doesn't work with the autoit lexer.

So now I want to write the lexer by myself. I've downloaded the scintilla source code and tried to build it. No problems there, I even tried changing stuff in the autoit lexer to see that everything worked.

Now comes the problems. Where do I start!? After going through the package I found python scripts updating the C code (oh that's never a good sign, code that updates code), enumerations and non-specific readme's that suffers from explaining detailed stuff very good but completely misses the basic "get started" points.

So what's the specific steps needed to get a extremely simple parser working? If I can get a custom lexer that only makes "// this is a comment" green I'm sure I could manage from there.

To my understanding this properties file is all that's needed to get the lexer working in SciTe:

# Skua is the name of my language
file.patterns.skua=*.skua;*.skua


filter.skua=Skua (skua)|$(file.patterns.skua)|

lexer.$(file.patterns.skua)=skua


# Comment
#style.skua.1=fore:#007F00,$(font.comment)

Thanks!

[Mat]

No idea... But will we be seeing the finished language ??

[monoceres]

No idea... But will we be seeing the finished language ??

Sure, it has some interesting features such as separate compilation, bytecode compilation and optimized recursion. But it depends on both ruby and java so I doubt many people will try it

[Jos]

Have you seen this page that describes the steps? : http://www.scintilla.org/SciTELexer.html

[monoceres]

Have you seen this page that describes the steps? : http://www.scintilla.org/SciTELexer.html

I have no idea how I managed to miss that page. It looks just what I want.

I'm sure I'll be back with more questions

Thanks!

[MvGulik]

whatever Edited February 7, 2011 by MvGulik

[monoceres]

Why not create a external scite lexer? (SciTEExternalLexer)

Would love to be able to create on myself for a particular language I'm playing around with. but ... o well.

That was my first thought. But it's just as much work and I have no problems building the entire package. Sticking with the original project is much easier.

[monoceres]

Ok, I'm really confused. As soon as I call styler.ColorTo() I crash scite.

What am I doing wrong here?

static void ColouriseSkuaDoc(unsigned int startPos, int length, int initStyle,WordList *keywordlists[],Accessor &styler) {


    int lengthDoc = startPos + length ;

    styler.ColourTo(lengthDoc - 1, SCE_SKUA_COMMENT);

}

Properties file looks like:

# Skua is the name of my language
file.patterns.skua=*.skua;*.skua


filter.skua=Skua (skua)|$(file.patterns.skua)|

lexer.$(file.patterns.skua)=skua

keywords.$(file.patterns.skua)=Func EndFunc

# Comment
style.skua.0=fore:#000000,$(font.comment)
style.skua.1=fore:#007F00,$(font.comment)

Also,

is it okay to take the easy way out and build the lexer with regular expressions?

Edited April 11, 2010 by monoceres

[monoceres]

I managed to clear it out. Adding:

StyleContext sc = StyleContext(startPos,startPos+length,SCE_SKUA_DEFAULT,styler);

Fixed it. In some way. I have no idea.

[monoceres]

Before I dive in any further - is it possible to add auto-intend without touching the lexer? I've seen the intend-auto directive but I have not been able to get it to work.

[Richard Robertson]

Do you perhaps mean indent?

[Valik]

Indentation is not controlled by the lexer, it's controlled by a series of properties. It can also be controlled by Lua (some of AutoIt's indentation is handled via Lua).

[monoceres]

Heh, I never seem to spell indentation right.

I don't get it though. What's wrong with my properties file?

expandcollapse popup

file.patterns.skua=*.skua;

filter.skua=Skua (skua)|$(file.patterns.skua)|

lexer.$(file.patterns.skua)=skua

keywords.$(file.patterns.skua)=FUNC ENDFUNC FOR IN ENDFOR IF ELSEIF ELSE ENDIF WHILE ENDWHILE LOCAL RETURN BREAK CONTINUE
keywords2.$(file.patterns.skua)=PRINT INPUT TOSTRING TYPEOF STRINGLENGTH ABSOLUTE STARTTIMER CALCTIMER RANGE SLEEP \
                                SQRT ALERT ARRAYLENGTH ARRAY CALL RANDOM 



statement.indent.$(file.patterns.skua)=5 Func EndFunc


tabsize=2
tab.size.$(file.patterns.skua)=2
indent.size=2
indent.size.$(file.patterns.skua)=2
use.tabs=1
use.tabs.$(file.patterns.skua)=1
indent.auto=1
tab.indents=2
backspace.unindents=2
indent.automatic=1
indent.opening=Func
indent.closing=EndFunc
indent.maintain.$(file.patterns.skua)=1
    

style.skua.0=fore:#000000
style.skua.1=fore:#007F00
style.skua.2=fore:#00007f,bold
style.skua.3=fore:#ee0000
style.skua.4=fore:#CC33FF
style.skua.5=fore:#006633,bold
style.skua.8=fore:#0080B5

command.go.*.skua=rubyw "$(SciteDefaultHome)\..\skuac.rb" "$(FileNameExt)"
command.build.*.skua=java -jar "$(SciteDefaultHome)\..\skuavm.jar" "$(FileName).sbc"

[MvGulik]

whatever Edited February 7, 2011 by MvGulik

[monoceres]

I don't get it though. What's wrong with my properties file?

In relation to ... what problem?

Getting auto-indentation to work. I wish scite to increase the indentation two tabs after a line that contains "Func".

[Valik]

You really need to read the SciTE documentation. The block.start and block.end properties are what control block indentation. Look at the AutoIt properties file to see how it is set up. Also... read the SciTE documentation about the properties.

[MvGulik]

whatever Edited February 7, 2011 by MvGulik

[Valik]

MvGulik, SciTE 2.10 has support for writing lexers in Lua. Any other (official) version of SciTE offering that functionality is out-of-date.

[MvGulik]

whatever Edited February 7, 2011 by MvGulik

[monoceres]

It's a bit easier to add functionality when you actually read the docs, instead of cutting pieces form other property files

Only auto-complete left, then it's magical!