feat: add full Unicode support for Javascript identifiers #3647

ctjlewis · 2020-07-21T02:30:38Z

Further resolves #3639, aims to prevent 99.999% of future issues that might occur from using non-English characters (except the few Greek and Latin chars manually included) in an identifier. In the current build, the following errors will occur (including a known issue with U+FF3F):

ERROR - [JSC_PARSE_ERROR] Parse error. Character '〩' (U+3029) is not a valid identifier start char
  1| var a〩 = 10;
          ^

ERROR - [JSC_PARSE_ERROR] Parse error. Character '〺' (U+303A) is not a valid identifier start char
  1| var 〺b = 10;
         ^

The caret is marked in the wrong spot for the last example, but the msg makes it clear that it's U+FF3F causing problems:

ERROR - [JSC_PARSE_ERROR] Parse error. Character '＿' (U+FF3F) is not a valid identifier start char
  1| var ａｅｓｔｈｅｔｉｃ＿;
                  ^

After:

var a〩 = 10;
var a\u3029=10;

var 〺b = 10;
var \u303ab=10;

var ａｅｓｔｈｅｔｉｃ＿;
var \uff41\uff45\uff53\uff54\uff48\uff45\uff54\uff49\uff43\uff3f;

Existing logic is retained for isIdentifierStart and isIdentifierPart, but if the character exists outside of the hardcoded ranges it is checked against regexpu-compiled patterns provided by TC39.

I would not expect that regexpu approach is very performant, but all previous optimizations were migrated into the UnicodeMatch class, so the only impact is if you use a previously unsupported character (which would have thrown an error and exited altogether prior to this patch). I will likely transform the matches into large (a <= ch & ch <= b) blocks to further optimize, but I wanted to put this up for review and feedback (especially regarding short-circuiting and branching conditions, which I am not super familiar with).

Apologies if this is an unwanted PR, I just figured I'd take a crack at it.

Update

I figured manually dumping a chain of bitwise operators would be faster than regexpu, but the compiled regex is consistently faster than the raw bitwise chain unless short-circuiting is used: https://repl.it/@christiantjl/UnicodeMatch

Jupyter notebook for generating the bitwise chain at: https://github.com/christiantjl/unicode-category-ranges/blob/master/Unicode.ipynb

…elying on java.lang.Character.

lauraharker · 2020-08-11T19:24:03Z

src/com/google/javascript/jscomp/parsing/parser/util/UnicodeMatch.java

+ * const regexIdentifierStart = /[$_\p{ID_Start}]/u;
+ * const regexIdentifierPart = /[$_\u200C\u200D\p{ID_Continue}]/u;
+ */
+public class UnicodeMatch {


Some visibility comments:

This class should be final. I think it shouldn't be public, unless we later find a compelling use case outside this package.

The Google Java best practices also recommends adding an empty private constructor for utility classes to make it clear they should not be instantiated, i.e.

private UnicodeMatch() {}

Thank you very much Laura, I will make the relevant changes.

If the class is made private, I can't import it inside Scanner.java - extremely sorry if this is basic syntax, I tried to research it, but is there any way to import just public isJavascriptIdentifierStart and public isJavascriptIdentifierPart from private UnicodeMatch without importing the whole class? I tried:

import com.google.javascript.jscomp.parsing.parser.util.UnicodeMatch.isJavascriptIdentifierStart; import com.google.javascript.jscomp.parsing.parser.util.UnicodeMatch.isJavascriptIdentifierPart;

But it will not build without importing the whole class (while public) via

import com.google.javascript.jscomp.parsing.parser.util.UnicodeMatch;

And referencing with UnicodeMatch.isJavascriptIdentifierStart and so on.

src/com/google/javascript/jscomp/parsing/parser/Scanner.java

lauraharker · 2020-08-11T19:36:44Z

src/com/google/javascript/jscomp/parsing/parser/util/UnicodeMatch.java

+public class UnicodeMatch {
+  // Intentional to minimize branches in this code
+  @SuppressWarnings("ShortCircuitBoolean")
+  private static boolean fastPathLetters(char ch) {


Could you also copy over the comment about minimizing branch prediction misses? It's not really clear to me otherwise why we are trying to minimize branches.

About adding the Latin and Greek letters to this fast path: I'm concerned that by adding those letters to this function, we're making the more common case of ASCII characters slower due to the lack of short-circuiting.

It seems potentially useful to have a fast path for these character, if they're common in Angular, but not at the expense of ASCII characters. @concavelenz was the one who initially added this fast path - John, do you have any thoughts?

We don't need to include non-ascii (>127) in the fast path.

So, to be clear, I should remove the non-ASCII part of the fast path (fast path effectively becomes just 0 < ch < 127)?

The Latin and Greek optimizations were there before I touched the code - will adjust, but just looking for acknowledgement that I'm not misunderstanding the direction, which may impact Angular build times. I did not insert those optimizations.

Edit: I misunderstood what the note was here, I believe I corrected the logic as needed.

src/com/google/javascript/jscomp/parsing/parser/Scanner.java

src/com/google/javascript/jscomp/parsing/parser/util/UnicodeMatch.java

concavelenz · 2020-08-11T22:33:29Z

src/com/google/javascript/jscomp/parsing/parser/util/UnicodeMatch.java

+      _: Character == '_'
+      \ UnicodeEscapeSequence: isEscapeSequence()
+  */
+  private static final Pattern JavascriptIdentifierStart = Pattern.compile("(?:[\\$A-Z_a-z\\xAA\\xB5\\xBA\\xC0-\\xD6\\xD8-\\xF6\\xF8-\\u02C1\\u02C6-\\u02D1\\u02E0-\\u02E4\\u02EC\\u02EE\\u0370-\\u0374\\u0376\\u0377\\u037A-\\u037D\\u037F\\u0386\\u0388-\\u038A\\u038C\\u038E-\\u03A1\\u03A3-\\u03F5\\u03F7-\\u0481\\u048A-\\u052F\\u0531-\\u0556\\u0559\\u0560-\\u0588\\u05D0-\\u05EA\\u05EF-\\u05F2\\u0620-\\u064A\\u066E\\u066F\\u0671-\\u06D3\\u06D5\\u06E5\\u06E6\\u06EE\\u06EF\\u06FA-\\u06FC\\u06FF\\u0710\\u0712-\\u072F\\u074D-\\u07A5\\u07B1\\u07CA-\\u07EA\\u07F4\\u07F5\\u07FA\\u0800-\\u0815\\u081A\\u0824\\u0828\\u0840-\\u0858\\u0860-\\u086A\\u08A0-\\u08B4\\u08B6-\\u08C7\\u0904-\\u0939\\u093D\\u0950\\u0958-\\u0961\\u0971-\\u0980\\u0985-\\u098C\\u098F\\u0990\\u0993-\\u09A8\\u09AA-\\u09B0\\u09B2\\u09B6-\\u09B9\\u09BD\\u09CE\\u09DC\\u09DD\\u09DF-\\u09E1\\u09F0\\u09F1\\u09FC\\u0A05-\\u0A0A\\u0A0F\\u0A10\\u0A13-\\u0A28\\u0A2A-\\u0A30\\u0A32\\u0A33\\u0A35\\u0A36\\u0A38\\u0A39\\u0A59-\\u0A5C\\u0A5E\\u0A72-\\u0A74\\u0A85-\\u0A8D\\u0A8F-\\u0A91\\u0A93-\\u0AA8\\u0AAA-\\u0AB0\\u0AB2\\u0AB3\\u0AB5-\\u0AB9\\u0ABD\\u0AD0\\u0AE0\\u0AE1\\u0AF9\\u0B05-\\u0B0C\\u0B0F\\u0B10\\u0B13-\\u0B28\\u0B2A-\\u0B30\\u0B32\\u0B33\\u0B35-\\u0B39\\u0B3D\\u0B5C\\u0B5D\\u0B5F-\\u0B61\\u0B71\\u0B83\\u0B85-\\u0B8A\\u0B8E-\\u0B90\\u0B92-\\u0B95\\u0B99\\u0B9A\\u0B9C\\u0B9E\\u0B9F\\u0BA3\\u0BA4\\u0BA8-\\u0BAA\\u0BAE-\\u0BB9\\u0BD0\\u0C05-\\u0C0C\\u0C0E-\\u0C10\\u0C12-\\u0C28\\u0C2A-\\u0C39\\u0C3D\\u0C58-\\u0C5A\\u0C60\\u0C61\\u0C80\\u0C85-\\u0C8C\\u0C8E-\\u0C90\\u0C92-\\u0CA8\\u0CAA-\\u0CB3\\u0CB5-\\u0CB9\\u0CBD\\u0CDE\\u0CE0\\u0CE1\\u0CF1\\u0CF2\\u0D04-\\u0D0C\\u0D0E-\\u0D10\\u0D12-\\u0D3A\\u0D3D\\u0D4E\\u0D54-\\u0D56\\u0D5F-\\u0D61\\u0D7A-\\u0D7F\\u0D85-\\u0D96\\u0D9A-\\u0DB1\\u0DB3-\\u0DBB\\u0DBD\\u0DC0-\\u0DC6\\u0E01-\\u0E30\\u0E32\\u0E33\\u0E40-\\u0E46\\u0E81\\u0E82\\u0E84\\u0E86-\\u0E8A\\u0E8C-\\u0EA3\\u0EA5\\u0EA7-\\u0EB0\\u0EB2\\u0EB3\\u0EBD\\u0EC0-\\u0EC4\\u0EC6\\u0EDC-\\u0EDF\\u0F00\\u0F40-\\u0F47\\u0F49-\\u0F6C\\u0F88-\\u0F8C\\u1000-\\u102A\\u103F\\u1050-\\u1055\\u105A-\\u105D\\u1061\\u1065\\u1066\\u106E-\\u1070\\u1075-\\u1081\\u108E\\u10A0-\\u10C5\\u10C7\\u10CD\\u10D0-\\u10FA\\u10FC-\\u1248\\u124A-\\u124D\\u1250-\\u1256\\u1258\\u125A-\\u125D\\u1260-\\u1288\\u128A-\\u128D\\u1290-\\u12B0\\u12B2-\\u12B5\\u12B8-\\u12BE\\u12C0\\u12C2-\\u12C5\\u12C8-\\u12D6\\u12D8-\\u1310\\u1312-\\u1315\\u1318-\\u135A\\u1380-\\u138F\\u13A0-\\u13F5\\u13F8-\\u13FD\\u1401-\\u166C\\u166F-\\u167F\\u1681-\\u169A\\u16A0-\\u16EA\\u16EE-\\u16F8\\u1700-\\u170C\\u170E-\\u1711\\u1720-\\u1731\\u1740-\\u1751\\u1760-\\u176C\\u176E-\\u1770\\u1780-\\u17B3\\u17D7\\u17DC\\u1820-\\u1878\\u1880-\\u18A8\\u18AA\\u18B0-\\u18F5\\u1900-\\u191E\\u1950-\\u196D\\u1970-\\u1974\\u1980-\\u19AB\\u19B0-\\u19C9\\u1A00-\\u1A16\\u1A20-\\u1A54\\u1AA7\\u1B05-\\u1B33\\u1B45-\\u1B4B\\u1B83-\\u1BA0\\u1BAE\\u1BAF\\u1BBA-\\u1BE5\\u1C00-\\u1C23\\u1C4D-\\u1C4F\\u1C5A-\\u1C7D\\u1C80-\\u1C88\\u1C90-\\u1CBA\\u1CBD-\\u1CBF\\u1CE9-\\u1CEC\\u1CEE-\\u1CF3\\u1CF5\\u1CF6\\u1CFA\\u1D00-\\u1DBF\\u1E00-\\u1F15\\u1F18-\\u1F1D\\u1F20-\\u1F45\\u1F48-\\u1F4D\\u1F50-\\u1F57\\u1F59\\u1F5B\\u1F5D\\u1F5F-\\u1F7D\\u1F80-\\u1FB4\\u1FB6-\\u1FBC\\u1FBE\\u1FC2-\\u1FC4\\u1FC6-\\u1FCC\\u1FD0-\\u1FD3\\u1FD6-\\u1FDB\\u1FE0-\\u1FEC\\u1FF2-\\u1FF4\\u1FF6-\\u1FFC\\u2071\\u207F\\u2090-\\u209C\\u2102\\u2107\\u210A-\\u2113\\u2115\\u2118-\\u211D\\u2124\\u2126\\u2128\\u212A-\\u2139\\u213C-\\u213F\\u2145-\\u2149\\u214E\\u2160-\\u2188\\u2C00-\\u2C2E\\u2C30-\\u2C5E\\u2C60-\\u2CE4\\u2CEB-\\u2CEE\\u2CF2\\u2CF3\\u2D00-\\u2D25\\u2D27\\u2D2D\\u2D30-\\u2D67\\u2D6F\\u2D80-\\u2D96\\u2DA0-\\u2DA6\\u2DA8-\\u2DAE\\u2DB0-\\u2DB6\\u2DB8-\\u2DBE\\u2DC0-\\u2DC6\\u2DC8-\\u2DCE\\u2DD0-\\u2DD6\\u2DD8-\\u2DDE\\u3005-\\u3007\\u3021-\\u3029\\u3031-\\u3035\\u3038-\\u303C\\u3041-\\u3096\\u309B-\\u309F\\u30A1-\\u30FA\\u30FC-\\u30FF\\u3105-\\u312F\\u3131-\\u318E\\u31A0-\\u31BF\\u31F0-\\u31FF\\u3400-\\u4DBF\\u4E00-\\u9FFC\\uA000-\\uA48C\\uA4D0-\\uA4FD\\uA500-\\uA60C\\uA610-\\uA61F\\uA62A\\uA62B\\uA640-\\uA66E\\uA67F-\\uA69D\\uA6A0-\\uA6EF\\uA717-\\uA71F\\uA722-\\uA788\\uA78B-\\uA7BF\\uA7C2-\\uA7CA\\uA7F5-\\uA801\\uA803-\\uA805\\uA807-\\uA80A\\uA80C-\\uA822\\uA840-\\uA873\\uA882-\\uA8B3\\uA8F2-\\uA8F7\\uA8FB\\uA8FD\\uA8FE\\uA90A-\\uA925\\uA930-\\uA946\\uA960-\\uA97C\\uA984-\\uA9B2\\uA9CF\\uA9E0-\\uA9E4\\uA9E6-\\uA9EF\\uA9FA-\\uA9FE\\uAA00-\\uAA28\\uAA40-\\uAA42\\uAA44-\\uAA4B\\uAA60-\\uAA76\\uAA7A\\uAA7E-\\uAAAF\\uAAB1\\uAAB5\\uAAB6\\uAAB9-\\uAABD\\uAAC0\\uAAC2\\uAADB-\\uAADD\\uAAE0-\\uAAEA\\uAAF2-\\uAAF4\\uAB01-\\uAB06\\uAB09-\\uAB0E\\uAB11-\\uAB16\\uAB20-\\uAB26\\uAB28-\\uAB2E\\uAB30-\\uAB5A\\uAB5C-\\uAB69\\uAB70-\\uABE2\\uAC00-\\uD7A3\\uD7B0-\\uD7C6\\uD7CB-\\uD7FB\\uF900-\\uFA6D\\uFA70-\\uFAD9\\uFB00-\\uFB06\\uFB13-\\uFB17\\uFB1D\\uFB1F-\\uFB28\\uFB2A-\\uFB36\\uFB38-\\uFB3C\\uFB3E\\uFB40\\uFB41\\uFB43\\uFB44\\uFB46-\\uFBB1\\uFBD3-\\uFD3D\\uFD50-\\uFD8F\\uFD92-\\uFDC7\\uFDF0-\\uFDFB\\uFE70-\\uFE74\\uFE76-\\uFEFC\\uFF21-\\uFF3A\\uFF41-\\uFF5A\\uFF66-\\uFFBE\\uFFC2-\\uFFC7\\uFFCA-\\uFFCF\\uFFD2-\\uFFD7\\uFFDA-\\uFFDC]|\\uD800[\\uDC00-\\uDC0B\\uDC0D-\\uDC26\\uDC28-\\uDC3A\\uDC3C\\uDC3D\\uDC3F-\\uDC4D\\uDC50-\\uDC5D\\uDC80-\\uDCFA\\uDD40-\\uDD74\\uDE80-\\uDE9C\\uDEA0-\\uDED0\\uDF00-\\uDF1F\\uDF2D-\\uDF4A\\uDF50-\\uDF75\\uDF80-\\uDF9D\\uDFA0-\\uDFC3\\uDFC8-\\uDFCF\\uDFD1-\\uDFD5]|\\uD801[\\uDC00-\\uDC9D\\uDCB0-\\uDCD3\\uDCD8-\\uDCFB\\uDD00-\\uDD27\\uDD30-\\uDD63\\uDE00-\\uDF36\\uDF40-\\uDF55\\uDF60-\\uDF67]|\\uD802[\\uDC00-\\uDC05\\uDC08\\uDC0A-\\uDC35\\uDC37\\uDC38\\uDC3C\\uDC3F-\\uDC55\\uDC60-\\uDC76\\uDC80-\\uDC9E\\uDCE0-\\uDCF2\\uDCF4\\uDCF5\\uDD00-\\uDD15\\uDD20-\\uDD39\\uDD80-\\uDDB7\\uDDBE\\uDDBF\\uDE00\\uDE10-\\uDE13\\uDE15-\\uDE17\\uDE19-\\uDE35\\uDE60-\\uDE7C\\uDE80-\\uDE9C\\uDEC0-\\uDEC7\\uDEC9-\\uDEE4\\uDF00-\\uDF35\\uDF40-\\uDF55\\uDF60-\\uDF72\\uDF80-\\uDF91]|\\uD803[\\uDC00-\\uDC48\\uDC80-\\uDCB2\\uDCC0-\\uDCF2\\uDD00-\\uDD23\\uDE80-\\uDEA9\\uDEB0\\uDEB1\\uDF00-\\uDF1C\\uDF27\\uDF30-\\uDF45\\uDFB0-\\uDFC4\\uDFE0-\\uDFF6]|\\uD804[\\uDC03-\\uDC37\\uDC83-\\uDCAF\\uDCD0-\\uDCE8\\uDD03-\\uDD26\\uDD44\\uDD47\\uDD50-\\uDD72\\uDD76\\uDD83-\\uDDB2\\uDDC1-\\uDDC4\\uDDDA\\uDDDC\\uDE00-\\uDE11\\uDE13-\\uDE2B\\uDE80-\\uDE86\\uDE88\\uDE8A-\\uDE8D\\uDE8F-\\uDE9D\\uDE9F-\\uDEA8\\uDEB0-\\uDEDE\\uDF05-\\uDF0C\\uDF0F\\uDF10\\uDF13-\\uDF28\\uDF2A-\\uDF30\\uDF32\\uDF33\\uDF35-\\uDF39\\uDF3D\\uDF50\\uDF5D-\\uDF61]|\\uD805[\\uDC00-\\uDC34\\uDC47-\\uDC4A\\uDC5F-\\uDC61\\uDC80-\\uDCAF\\uDCC4\\uDCC5\\uDCC7\\uDD80-\\uDDAE\\uDDD8-\\uDDDB\\uDE00-\\uDE2F\\uDE44\\uDE80-\\uDEAA\\uDEB8\\uDF00-\\uDF1A]|\\uD806[\\uDC00-\\uDC2B\\uDCA0-\\uDCDF\\uDCFF-\\uDD06\\uDD09\\uDD0C-\\uDD13\\uDD15\\uDD16\\uDD18-\\uDD2F\\uDD3F\\uDD41\\uDDA0-\\uDDA7\\uDDAA-\\uDDD0\\uDDE1\\uDDE3\\uDE00\\uDE0B-\\uDE32\\uDE3A\\uDE50\\uDE5C-\\uDE89\\uDE9D\\uDEC0-\\uDEF8]|\\uD807[\\uDC00-\\uDC08\\uDC0A-\\uDC2E\\uDC40\\uDC72-\\uDC8F\\uDD00-\\uDD06\\uDD08\\uDD09\\uDD0B-\\uDD30\\uDD46\\uDD60-\\uDD65\\uDD67\\uDD68\\uDD6A-\\uDD89\\uDD98\\uDEE0-\\uDEF2\\uDFB0]|\\uD808[\\uDC00-\\uDF99]|\\uD809[\\uDC00-\\uDC6E\\uDC80-\\uDD43]|[\\uD80C\\uD81C-\\uD820\\uD822\\uD840-\\uD868\\uD86A-\\uD86C\\uD86F-\\uD872\\uD874-\\uD879\\uD880-\\uD883][\\uDC00-\\uDFFF]|\\uD80D[\\uDC00-\\uDC2E]|\\uD811[\\uDC00-\\uDE46]|\\uD81A[\\uDC00-\\uDE38\\uDE40-\\uDE5E\\uDED0-\\uDEED\\uDF00-\\uDF2F\\uDF40-\\uDF43\\uDF63-\\uDF77\\uDF7D-\\uDF8F]|\\uD81B[\\uDE40-\\uDE7F\\uDF00-\\uDF4A\\uDF50\\uDF93-\\uDF9F\\uDFE0\\uDFE1\\uDFE3]|\\uD821[\\uDC00-\\uDFF7]|\\uD823[\\uDC00-\\uDCD5\\uDD00-\\uDD08]|\\uD82C[\\uDC00-\\uDD1E\\uDD50-\\uDD52\\uDD64-\\uDD67\\uDD70-\\uDEFB]|\\uD82F[\\uDC00-\\uDC6A\\uDC70-\\uDC7C\\uDC80-\\uDC88\\uDC90-\\uDC99]|\\uD835[\\uDC00-\\uDC54\\uDC56-\\uDC9C\\uDC9E\\uDC9F\\uDCA2\\uDCA5\\uDCA6\\uDCA9-\\uDCAC\\uDCAE-\\uDCB9\\uDCBB\\uDCBD-\\uDCC3\\uDCC5-\\uDD05\\uDD07-\\uDD0A\\uDD0D-\\uDD14\\uDD16-\\uDD1C\\uDD1E-\\uDD39\\uDD3B-\\uDD3E\\uDD40-\\uDD44\\uDD46\\uDD4A-\\uDD50\\uDD52-\\uDEA5\\uDEA8-\\uDEC0\\uDEC2-\\uDEDA\\uDEDC-\\uDEFA\\uDEFC-\\uDF14\\uDF16-\\uDF34\\uDF36-\\uDF4E\\uDF50-\\uDF6E\\uDF70-\\uDF88\\uDF8A-\\uDFA8\\uDFAA-\\uDFC2\\uDFC4-\\uDFCB]|\\uD838[\\uDD00-\\uDD2C\\uDD37-\\uDD3D\\uDD4E\\uDEC0-\\uDEEB]|\\uD83A[\\uDC00-\\uDCC4\\uDD00-\\uDD43\\uDD4B]|\\uD83B[\\uDE00-\\uDE03\\uDE05-\\uDE1F\\uDE21\\uDE22\\uDE24\\uDE27\\uDE29-\\uDE32\\uDE34-\\uDE37\\uDE39\\uDE3B\\uDE42\\uDE47\\uDE49\\uDE4B\\uDE4D-\\uDE4F\\uDE51\\uDE52\\uDE54\\uDE57\\uDE59\\uDE5B\\uDE5D\\uDE5F\\uDE61\\uDE62\\uDE64\\uDE67-\\uDE6A\\uDE6C-\\uDE72\\uDE74-\\uDE77\\uDE79-\\uDE7C\\uDE7E\\uDE80-\\uDE89\\uDE8B-\\uDE9B\\uDEA1-\\uDEA3\\uDEA5-\\uDEA9\\uDEAB-\\uDEBB]|\\uD869[\\uDC00-\\uDEDD\\uDF00-\\uDFFF]|\\uD86D[\\uDC00-\\uDF34\\uDF40-\\uDFFF]|\\uD86E[\\uDC00-\\uDC1D\\uDC20-\\uDFFF]|\\uD873[\\uDC00-\\uDEA1\\uDEB0-\\uDFFF]|\\uD87A[\\uDC00-\\uDFE0]|\\uD87E[\\uDC00-\\uDE1D]|\\uD884[\\uDC00-\\uDF4A])");


How did you generate these values?

These are the regexpu-transpiled patterns provided by TC39: https://github.com/tc39/proposal-regexp-unicode-property-escapes#other-examples

const regexIdentifierStart = /[$_\p{ID_Start}]/u; const regexIdentifierPart = /[$_\u200C\u200D\p{ID_Continue}]/u;

Online regexpu portal at: https://mothereff.in/regexpu

See also the Updates section of the OP and the JSDOC comments in this class.

concavelenz · 2020-08-11T22:34:41Z

src/com/google/javascript/jscomp/parsing/parser/Scanner.java

-
-    // Handle non-ASCII characters.
-    // TODO(tjgq): This should include all characters with the ID_Start property.
-    if (Character.isLetter(ch)) {


Do you feel this is insufficient for the native Java builds? Where do you believe you need the JavaScript version?

@concavelenz I believe the reason I removed Character.isLetter was because the GWT build would not work for non-ASCII unicode, due to limitations of J2CL / Guava implementations of those methods. See: google/j2cl#103

Apologies for the lack of specificity, this issue is about a month old and is not super fresh in my mind. Thank you for your reviews, John.

src/com/google/javascript/jscomp/parsing/parser/util/UnicodeMatch.java

… int

ctjlewis · 2020-08-13T00:00:09Z

src/com/google/javascript/jscomp/parsing/parser/util/UnicodeMatch.java

+    if (ch <= 127) {
+      return (
+        (ch >= 'A' & ch <= 'Z') |
+        (ch >= 'a' & ch <= 'z') |
+        (ch == '$' | ch == '_')
+      );
+    }


ASCII fast path will work in every call to fastPathIdentifierStart and fastPathIdentifierPart (because the former is evaluated in the latter).

rishipal · 2020-10-29T21:21:37Z

This isn't merged yet. I'm testing this internally and will update soon.

rishipal · 2020-11-04T04:42:30Z

I'm seeing multiple internal targets failing to build with this error message - 'The import java.util.regex cannot be resolved'.

rishipal · 2020-12-11T17:43:50Z

I'll take another stab at this today and repro+debug the failing tests.

ctjlewis · 2020-12-11T17:45:04Z

Hey @rishipal, I saw your comment about errors, I did not see any when I submitted the PR. Apologies for the delay, I have not had much free time to review this.

rishipal · 2020-12-14T17:34:02Z

The 'The import java.util.regex cannot be resolved' failures get fixed after switching to re2j regex library. However, I see multiple Scanner tests failing with that. I will update shortly.

concavelenz · 2021-02-11T20:45:57Z

We have dropped support for the JS (GWT version) of the Closure Compiler in favor of the Graal native image versions. Do you feel that this is still useful?

ctjlewis · 2021-02-14T22:45:50Z

@concavelenz As I understand it, this was an issue regardless of build - it was just an issue of extended Unicode characters not accurately being counted as valid identifier parts, IIRC (this work is not super fresh in my mind, apologies).

rishipal · 2021-02-16T17:19:29Z

Update: I tested this again over the weekend. The internal tests are passing now. I will commit this internally.

ctjlewis · 2021-03-05T06:54:22Z

@rishipal Thank you for your help on this PR. Was this merged internally?

Sorry to bother!

rishipal · 2021-03-10T01:19:58Z

Hi, this is not merged yet. A comment from John on the pending CL suggests that this may not be needed after we stopped support for GWT/J2CL version of the compiler. I will confirm the priority of this and respond by EOD tomorrow.

rishipal · 2021-03-10T17:27:04Z

The Scanner test failures did not get fully resolved contrary to my #3647 (comment). This PR is decided to be deprioritized and shelved.

ctjlewis added 12 commits July 19, 2020 17:47

Added support for UnicodeCombiningMark, fixes google#3639.

5ad0744

Removed unnecessary test script, moved scripts to utils/.

93ed59e

Added newlines for formatting

7d2ea29

Patched isCombiningMark to check Unicode space manually rather than r…

3c5ee21

…elying on java.lang.Character.

Added bazel output symlinks to .gitignore and removed from tree.

b3a720c

Removed unneeded java.lang.Character import.

590f2dc

Removed unneeded test.js.

1ecfd75

Merge remote-tracking branch 'upstream/master'

e1d5129

Adding support for non-ASCII Unicode.

696e201

Added .vscode to .gitignore.

0d25c89

Added note about regexpu.

f2f02e3

Fixed test re. U+FF3F, which is now supported.

a8e9e95

googlebot added the cla: yes label Jul 21, 2020

Added more notes on regex patterns used.

813797b

ctjlewis mentioned this pull request Jul 21, 2020

Added support for UnicodeCombiningMark, fixes #3639. #3645

Closed

rishipal self-assigned this Jul 22, 2020

ctjlewis marked this pull request as ready for review July 22, 2020 03:22

ctjlewis mentioned this pull request Aug 9, 2020

Improve unicode escape in regex #3656

Merged

lauraharker reviewed Aug 11, 2020

View reviewed changes

concavelenz reviewed Aug 11, 2020

View reviewed changes

src/com/google/javascript/jscomp/parsing/parser/Scanner.java Outdated Show resolved Hide resolved

concavelenz reviewed Aug 11, 2020

View reviewed changes

src/com/google/javascript/jscomp/parsing/parser/Scanner.java Outdated Show resolved Hide resolved

concavelenz reviewed Aug 11, 2020

View reviewed changes

src/com/google/javascript/jscomp/parsing/parser/util/UnicodeMatch.java Outdated Show resolved Hide resolved

concavelenz reviewed Aug 11, 2020

View reviewed changes

tjgq reviewed Aug 12, 2020

View reviewed changes

src/com/google/javascript/jscomp/parsing/parser/util/UnicodeMatch.java Outdated Show resolved Hide resolved

ctjlewis added 4 commits August 12, 2020 14:37

Inlined calls to UnicodeMatch utils.

8fefd7d

UnicodeMatch to public class, prevent build failure

78700f2

Merge branch 'master' into unicode

da68979

Changed isIdentifierPart and isIdentifierStart arg types from char to…

ad71a90

… int

ctjlewis added 12 commits August 12, 2020 16:05

Clarified comments and added fastPathAscii

066eaba

Re-added fastPath Greek/Latin checks until further advisement

8c7c040

Reverting fastPathAscii check until further advisement

72d6ce8

Refactored fast paths

7e83d4f

Removed debugging utils/ directory

2f69388

Comment formatting

dc275d2

Fixed string/char error

bd26958

Comment formatting

5fb993f

Committing changes before push to remote.

8ab20b7

Explicit typecast to char for Character.toString()

cb50fe1

Updated fast paths for ASCII optimization

469265f

Comment formatting

6d9f04e

ctjlewis requested review from concavelenz, lauraharker and tjgq August 12, 2020 23:57

ctjlewis commented Aug 13, 2020

View reviewed changes

rishipal removed their assignment Jun 3, 2021

ctjlewis changed the title ~~Adding full Unicode support for Javascript identifiers.~~ feat: add full Unicode support for Javascript identifiers. Jul 18, 2021

ctjlewis changed the title ~~feat: add full Unicode support for Javascript identifiers.~~ feat: add full Unicode support for Javascript identifiers Jul 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add full Unicode support for Javascript identifiers #3647

feat: add full Unicode support for Javascript identifiers #3647

ctjlewis commented Jul 21, 2020 •

edited

lauraharker Aug 11, 2020

ctjlewis Aug 12, 2020

ctjlewis Aug 12, 2020 •

edited

lauraharker Aug 11, 2020

concavelenz Aug 11, 2020

ctjlewis Aug 12, 2020

ctjlewis Aug 12, 2020 •

edited

concavelenz Aug 11, 2020

ctjlewis Aug 12, 2020 •

edited

concavelenz Aug 11, 2020

ctjlewis Aug 12, 2020 •

edited

ctjlewis Aug 13, 2020

rishipal commented Oct 29, 2020

rishipal commented Nov 4, 2020

rishipal commented Dec 11, 2020

ctjlewis commented Dec 11, 2020

rishipal commented Dec 14, 2020

concavelenz commented Feb 11, 2021

ctjlewis commented Feb 14, 2021

rishipal commented Feb 16, 2021

ctjlewis commented Mar 5, 2021

rishipal commented Mar 10, 2021

rishipal commented Mar 10, 2021

feat: add full Unicode support for Javascript identifiers #3647

Are you sure you want to change the base?

feat: add full Unicode support for Javascript identifiers #3647

Conversation

ctjlewis commented Jul 21, 2020 • edited

Update

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctjlewis Aug 12, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctjlewis Aug 12, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctjlewis Aug 12, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctjlewis Aug 12, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rishipal commented Oct 29, 2020

rishipal commented Nov 4, 2020

rishipal commented Dec 11, 2020

ctjlewis commented Dec 11, 2020

rishipal commented Dec 14, 2020

concavelenz commented Feb 11, 2021

ctjlewis commented Feb 14, 2021

rishipal commented Feb 16, 2021

ctjlewis commented Mar 5, 2021

rishipal commented Mar 10, 2021

rishipal commented Mar 10, 2021

ctjlewis commented Jul 21, 2020 •

edited

ctjlewis Aug 12, 2020 •

edited

ctjlewis Aug 12, 2020 •

edited

ctjlewis Aug 12, 2020 •

edited

ctjlewis Aug 12, 2020 •

edited