Proposal: embedded syntax block

Anders Hejlsberg (@ahejlsberg)

I have watched this PR #2673 and this #296 discussion for a while. But I somehow don't think it is the right approach to accept JSX directly into TS. There seem to be a demand of embedding HTML and even CSS in JS. Facebook might be the pioneer in embedding HTML in JS and therefore gotten a lot of attention recently with JSX/React. If JSX got accepted into TS, it would probably send a clear message to the JS and TS community that the TS team thinks JSX is the right way(and the only way) to go in terms of rendering views. I think it is more a correct way to accept an "embedded syntax block", so different frameworks have a chance to compete with JSX. It is a more generalized approach as oppose to accepting JSX directly into the TS.

Also remember that Anders Hejlsberg (@ahejlsberg) said "A framework is created nearly every day". So who knows if React will survive the next 2 years? Just remember how fast Angular 1 was popular and very quickly perceived as outdated(roughly a year?). That's how fast the web moves today.

So in rough terms here is my thoughts on this "embedded syntax block". I will take JSX as an example:

First, we refer to a JSX to TS transpiler. So we can transpile JSX to TS in our source code:

///<transpiler name="jsx" path="../node_modules/jsx-ts-transpiler/transpiler.ts"/>

Now, we define a JSX block we want transpiled into TS using the syntax syntax 'jsx' {...}:

let HelloMessage = React.createClass({
  render: function() {
    return syntax 'jsx' { <div>Hello {this.props.name}</div> }
  }
});

The TS's scanner stops at syntax 'jsx' { and feeds the string that reaches the closing brace } of the syntax block to the JSX-TS transpiler in this case: <div>Hello {this.props.name}</div>. The JSX-TS transpiler emits the correct TS code back along with a source map so we can map any lines and columns in the emitted TS back to our JSX. Here is how the transpiler looks like:

export transpile(source: string): Output {
  // Do something with the source
  return { output, sourceMap, diagnostics };
}

Now the TS scanner goes ahead an parse it as if it was a normal TS file.

let HelloMessage = React.createClass({
  render: function() {
    return React.createElement("div", null, "Hello", this.props.name);
  }
});

Now, we want all AST nodes outside of any "embedded syntax block" to have the same positions(pos and end) as if they where without any line and column shifts. So that we can have the correct positions on errors and warnings autocomplete etc. This is easily done by just offsetting a node's pos and end by keeping track of how much the emitted JSX-TS output shifts in character positions. For the emitted AST nodes we would marked them coming from a particular embedded syntax block(the embedded syntax block itself is an AST node with pos and end). The emitted TS nodes will be assigned pos and end beginning from 0 and not the start of its' syntax block.

Now, we have all the information we need to output the correct positions in source map and diagnostics. The diagnostics and the source map needs to be rewritten to take into consideration the new embedded syntax blocks along with its' emitted TS code.

I think this is a more open approach. If you accepts JSX directly into TS, then you are practically closing the door for any future syntax embeds and letting Facebook decide the future of syntax embeds.

Updates 1:

Since the TS scanner can't decide where a syntax embed block ends we would need the transpiler to help us so here is the updated transpiler:

let output = "";
let sourceMap;
let diagnostics;
let statements;

export function pushChar(ch: number): boolean {
  // Do something that produces an AST

  // Return false if it is the end of a syntax embed block else return true.
  return true;
}

export function transpile(): Output {
  // Take the AST and produce the output, source map and diagnostics.
  return { output, sourceMap, diagnostics};
}

The TS scanner pushes characters to our transpiler with pushChar() and receives true or false, whether to proceed with pushing characters to our transpiler or get the transpiled output with transpile().

The syntax for defining syntax embed could drop the syntax keyword:

'jsx' {  <div>Hello {{this.props.name}}</div> }

How would the transformer do more complicated transformations that just string replacements on the source string? For example, how would it do a transformation that's dependent on the types of the things in the string that it's transforming, if all it has access to is the string itself?
Even parsing the string becomes non-trivial when you consider a transform that supports embedded JS / TS (such as JSX). Then the transform function needs to invoke a full parser, perhaps invoke the TS parser in some sort of nested context.
I guess with your proposal that type-checking, completion information, etc. will be provided by TS on the transformed code, and not by the transformer itself.
Do braces inside the string need to be balanced? That is, can the DSL inside the syntax block have a closing brace without an open brace and not confuse the TS parser into thinking the syntax block ends earlier than it actually does?

How would the transformer do more complicated transformations that just string replacements on the source string? For example, how would it do a transformation that's dependent on the types of the things in the string that it's transforming, if all it has access to is the string itself?

It's meant to transforms one string to an another. So a perfect use case is embedding HTML templates or creating your own CSS language that can live inline with your TS code. But instead of writing it in strings("..."), which React team and others like me thinks it creates "syntax pollution". Or writing it on a separate file you would need to create a separate build and tape things together which is quite a hell.

It isn't meant to be a "full" programming language to "full" programming language embed. So it isn't meant that you can embed PHP code. There is no benefit of embedding a full programming language than to write TS code alone. JSX described itself as "XML-like syntax extension to EcmaScript" — and that is what this embed functionality is meant for extending the TS language with additional syntaxes.

Even parsing the string becomes non-trivial when you consider a transform that supports embedded JS / TS (such as React). Then the transform function needs to invoke a full parser, perhaps invoke the TS parser in some sort of nested context.

The proposed embedded syntax block is only functioning as sugar. If the embed already contains TS code that are marked(in JSX it is marked inside { ... }). The one who writes the transpiler need to make sure that code isn't transformed. The transpiler doesn't need to understand TS code — just where it is. Even a trivial example let element = <MyComponent/> we can figure out what is TS code and what is JSX.

I guess with your proposal that type-checking, completion information, etc. will be provided by TS on the transformed code, and not by the transformer itself.

yes.

Do braces inside the string need to be balanced? That is, can the DSL inside the syntax block have a closing brace without an open brace and not confuse the TS parser into thinking the syntax block ends earlier than it actually does?

I missed this in my proposal. If balancing works then yes.

The transpiler doesn't need to understand TS code — just where it is.

My point is that it does. To take your example:

syntax 'jsx' { <div>Hello {this.props.name}</div> }

I assume that this will be passed as the string "<div>Hello {this.props.name}</div>" to transpile(). Then transpile() will need to interpret this string to realize that it consists of a tag "div", that then contains the string "Hello " and a TS expression "this.props.name", and then generate the appropriate codegen in your OP.

Now consider if this TS expression contains braces, double quotes, escapes, etc. Now transpile() has to implement some of the TS parser to be able to make sense of this embedded DSL. In fact, there can be more than one such {...} expression and you need to know where one ends and the next begins.

syntax 'jsx' { <div>Hello { `this.props.name${ "I'm a template string expression which has its own braces
and contains newlines. I could even contain entire JS statements inside IIFEs." } { This is an unterminated open brace` } }</div> }

transpile() will have a hard time parsing this and converting it into real TS without being able to invoke the TS parser on it.

What you're essentially describing is a very unrestricted pluggable parser-emitter, which I'm not sure is doable with a grammar such as TS. Compare this to, say, sweet.js, where the unit of transformation for the macro engine is pre-parsed tokens that vaguely resemble JS, not arbitrary text.

Now consider if this TS expression contains braces, double quotes, escapes, etc. Now transpile() has to implement some of the TS parser to be able to make sense of this embedded DSL. In fact, there can be more than one such {...} expression and you need to know where one ends and the next begins.

It was a rough proposal 😄 . But one can probably make the TS evaluated only inside {{...}} or any syntax that is very distinguished from TS, in worst case even add escape characters and let the transpiler unescape them.

Instead of having balanced braces for calculating the end of the embed. All characters are feed one-by-one to a scanner/parser of the transpiler including the open brace { and ending brace } of the embed and the scanner/parser returns if is the end of the embed or not for every character.

///<transpiler name="jsx" path="../node_modules/jsx-ts-transpiler/transpiler.ts"/>

Perhaps better registered as transpilers using tsconfig.json.

But one can probably make the TS evaluated only inside {{...}} or any syntax that is very distinguished with TS

Yep, exactly my point. That is, your transpile() isn't a function from source string to TS. Rather, the TS parser needs to first split the text inside the syntax block into strings and expressions, and transpile() would get such a sequence of strings and expressions that it can then compose into the final TS codegen.

(And that should sound a lot like tagged template strings, because it is. Although template string tag functions deal with the results of embedded expressions, not the expressions themselves; but it's the same in that the parser needs to be aware of how to parse them and the tagger needs to work on the separate components. Of course, the value here is to retain the existing DSL rather than have to convert it to template strings.)

Yep, exactly my point. That is, your transpile() isn't a function from source string to TS. Rather, the TS parser needs to first split the text inside the syntax block into strings and expressions, and transpile() would get such a sequence of strings and expressions that it can then compose into the final TS codegen.

Just for clarifications, there is no need for splits? I meant {{...}} is for TS code inside JSX and then feed every character to the JSX-TS scanner/parser and the scanner/parser returns if it is in the end of the syntax embed or not.

So an end can be calculated even if we have(no need to involve the TS parser):

syntax 'jsx' { <div>Hello {{`hello world :{  ${message`}}</div> }

The above example will generate this invalid TS code(from valid JSX code):

React.createElement('div', null, 'Hello' , `hello world :{ ${message`);

No matter how complicated you make your delimiter, it'll always be possible to use it in a way where it doesn't function as a delimiter.

syntax 'jsx' { <div>Hello {{ "hello world }} {{" }}</div> }

consists of a string "Hello" and a single expression - the string "hello world }} {{". A naive algorithm that doesn't understand TS syntax and only looks for the delimiter would instead infer two expressions.

As mentioned earlier, you would need to escape }} {{, though having {{ and }} makes escaping happening rarely as oppose to just having { and }.

syntax 'jsx' { <div>Hello {{ "hello world \}\} \{\{" }}</div> }

The transpiler will then unescape those:

React.createElement('div', null, 'Hello' , "hello world }} {{");

Perhaps better registered as transpilers using tsconfig.json.

True, I leave it up the TS team to decide if they want to go along this road. Then I guess, it also need to have a command line option.

Side note: if possible perhaps having the shorter syntax 'transpiler' {...} by dropping the syntax keyword. So for JSX:

'jsx' {  <div>Hello {{this.props.name}}</div> }

I think a solution that didn't add any extra syntax to the language, except for the DSL that is embedded, would be much cleaner (something like Babel's transformers).

Chris Prucha (@icetraxx)

Chris Prucha (@icetraxx) isn't 'jsx' {...} small enough? You would need at least a symbol for the start {, and one for the end } and if you want to support more than one type of embed you need to specify the type 'jsx' of the embed?

For the Babel transforms:
I'm not sure if this is correct but it seems like Babel are visiting/looping through all the nodes one per custom transform and is only editable within their range of supported AST nodes?
http://babeljs.io/docs/usage/plugins/

This solutions doesn't create an additional loop and doesn't rely on TS team to support any new AST nodes.

the competing frameworks have even wider support and stronger adoption.

Can you be more specific? Which of these extend JS syntax?

Knockout for example defines its logic in the data-bind attribute. This is currently embedded in the HTML, but the proposal above would have brought it into the JavaScript domain.

You also forgot to mention all template languages Mustasch, Handlebars even maybe Angular? could have an interest on this.

JSX is just a new kid on the block fed with a silver spoon.

I am okay with the TS team adding native JSX support specific to React 🌹. Still curious about non react implementations though e.g. https://github.com/vjeux/jsxdom this is a fundamental motivation behind the jsx spec : https://facebook.github.io/jsx/#transpilers < is this out of scope as well or needs a different proposal?

Also notice that Handlebars has 10x more downloads than React on NPM:

Charts for Handlebars:
http://npm-stat.com/charts.html?package=handlebars&author=&from=2013-05-25&to=2015-05-25
Charts for React:
http://npm-stat.com/charts.html?package=react&author=&from=&to=

Also notice that Handlebars has 10x more downloads than React on NPM:

Not that I am concerned about this specific conversations, but making architectural decisions based on the number of downloads somethings has on NPM is quite ill advised logic.

I definitely agree with you, but wasn't it the numbers that made the TS team include JSX in the first place? And just if I can speculate(which is bad) added the postfix type assertion operator as just to support JSX?

Tingan Ho (@tinganho)

Tingan Ho (@tinganho) worth your review : TypeStrong/atom-typescript#362

Basarat Ali Syed (@basarat)

Basarat Ali Syed (@basarat) Atom-typescript is a great plugin. But I think this feature should be implemented in TS for various reason, one of them being that not everyone is using the same IDE. Another reason is that this problem belongs to the TS compiler, so it would be best if they could fix it.

Ryan Cavanaugh (@RyanCavanaugh) how about instead of an embed having pluggable syntax support? Acorn have already re-written its parser to support syntax plugins. The current TS JSX implementation could then be rewritten to use it? Though it would require some hooks to be implemented in the scanner, parser emitter checker etc.

It looks very nice in their example code:
https://github.com/marijnh/acorn/blob/jsx/plugin/jsx.js#L608-L662

Relevant info:
https://github.com/marijnh/acorn#plugins

Relevant discussion:
acornjs/acorn#168

I'm thinking:

syntax:jsx{ 
      /* code here */ 
}syntax:jsx

Making the need for any escaping very unlikely. This is very much like <span> </span> => syntax{ }syntax and easy to parse and partition out.

Basarat Ali Syed (@basarat)

Basarat Ali Syed (@basarat), I like what you are trying to do.

Here is another syntax that turned up in my research, Rust uses syntaxName! { } e.g. Maud which is a syntax extension for html : html! { } : http://lfairy.gitbooks.io/maud/content/basic_syntax.html

I still prefer my stronger delimiters

Basarat Ali Syed (@basarat)

Basarat Ali Syed (@basarat) turns out that Rust already has an syntax extension API https://doc.rust-lang.org/book/compiler-plugins.html.

Tingan Ho (@tinganho)

turns out that Rust already has an syntax extension API

Tingan Ho (@tinganho) Yup. That's what I meant with "Rust uses syntaxName! {" Thanks for the link though, I should have added that 👍

Navigation Menu

Repository navigation

Proposal: embedded syntax block #3022

Updates 1:

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: embedded syntax block #3022

Description

Updates 1:

Activity

Arnavion commented on May 5, 2015

tinganho commented on May 5, 2015

Arnavion commented on May 5, 2015

tinganho commented on May 5, 2015

basarat commented on May 5, 2015

Arnavion commented on May 5, 2015

tinganho commented on May 5, 2015

Arnavion commented on May 5, 2015

tinganho commented on May 5, 2015

tinganho commented on May 5, 2015

tinganho commented on May 5, 2015

icetraxx commented on May 5, 2015

tinganho commented on May 5, 2015

21 remaining items

RyanCavanaugh commented on May 20, 2015

NoelAbrahams commented on May 20, 2015

tinganho commented on May 24, 2015

basarat commented on May 25, 2015

tinganho commented on May 25, 2015

kitsonk commented on May 25, 2015

tinganho commented on May 25, 2015

basarat commented on May 26, 2015

tinganho commented on May 26, 2015

basarat commented on May 27, 2015

NoelAbrahams commented on May 27, 2015

basarat commented on Jun 1, 2015

tinganho commented on Jun 1, 2015

basarat commented on Jun 1, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions