web: replace rollup with esbuild (#8699)
* Holding for a moment... * web: replace rollup with esbuild This commit replaces rollup with esbuild. The biggest fix was to alter the way CSS is imported into our system; esbuild delivers it to the browser as text, rather than as a bundle with metadata that, frankly, we never use. ESBuild will bundle the CSS for us just fine, and interpreting those strings *as* CSS turned out to be a small hurdle. Code has been added to AKElement and Interface to ensure that all CSS referenced by an element has been converted to a Browser CSSStyleSheet before being presented to the browser. A similar fix has been provided for the markdown imports. The biggest headache there was that the re-arrangement of our documentation broke Jen's existing parser for fixing relative links. I've provided a corresponding hack that provides the necessary detail, but since the Markdown is being presented to the browser as text, we have to provide a hint in the markdown component for where any relative links should go, and we're importing and processing the markdown at runtime. This doesn't seem to be a big performance hit. The entire build process is driven by the new build script, `build.mjs`, which starts the esbuild process as a service connected to the build script and then runs the commands sent to it as fast as possible. The biggest "hack" in it is actually the replacement for rollup's `rollup-copy-plugin`, which is clever enough I'm surprised it doesn't exist as a standalone file-copy package in its own right. I've also used a filesystem watch library to encode a "watcher" mechanism into the build script. `node build.mjs --watch` will work on MacOS; I haven't tested it elsewhere, at least not yet. `node build.mjs --proxy` does what the old rollup.proxy.js script did. The savings are substantial. It takes less than two seconds to build the whole UI, a huge savings off the older ~45-50 seconds I routinely saw on my old Mac. It's also about 9% smaller. The trade-offs appear to be small: processing the CSS as StyleSheets, and the Markdown as HTML, at run-time is a small performance hit, but I didn't notice it in amongst everything else the UI does as it starts up. Manual chunking is gone; esbuild's support for that is quite difficult to get right compared to Rollup's, although there's been a bit of yelling at ESbuild over it. Codemirror is built into its own chunk; it's just not _named_ distinctly anymore. The one thing I haven't been able to test yet is whether or not the polyfills and runtim shims work as expected on older browsers. * web: continue with performance and build fixes This commit introduces a couple of fixes enabled by esbuild and other features. 1. build-locales `build-locales` is a new NodeJS script in the `./scripts` folder that does pretty much what it says in the name: it translates Xliff files into `.ts` files. It has two DevExp advantages over the old build system. First, it will check the build times of the xlf files and their ts equivalents, and will only run the actual build-locales command if the XLF files are newer than their TS equivalents. Second, it captures the stderr output from the build-locales command and summarizes it. Instead of the thousands of lines of "this string has no translation equivalent," now it just reports the number of missed translations per locale. 2. check-spelling This is a simple wrapper around the `codespell` command, mostly just to reduce the visual clutter of `package.json`, but also to permit it to run just about anywhere without needed hard-coded paths to the dictionaries, using a fairly classic trick with git. 3. pseudolocalize and import-maps These scripts were in TypeScript, but for our purposes I've saved their constructed equivalents instead. This saves on visual clutter in the `package.json` script, and reduced the time they have to run during full builds. They're small enough I feel confident they won't need too much looking over. Also, two lint bugs in Markdown.ts have been fixed. * Removed a few lines that weren't in use. * build-locales was sufficiently complex it needed some comments. * web: formalize that horrible unixy git status checker into a proper function. * Added types for , the Markdown processor for in-line documentation. * re-add dependencies required for storybook Signed-off-by: Jens Langhammer <jens@goauthentik.io> * fix optional deps Signed-off-by: Jens Langhammer <jens@goauthentik.io> * fix relative links for docs Signed-off-by: Jens Langhammer <jens@goauthentik.io> * only build once on startup Signed-off-by: Jens Langhammer <jens@goauthentik.io> * prevent crash when build fails in watch mode, improve console output Signed-off-by: Jens Langhammer <jens@goauthentik.io> --------- Signed-off-by: Jens Langhammer <jens@goauthentik.io> Co-authored-by: Jens Langhammer <jens@goauthentik.io>
This commit is contained in:
Ken Sternberg
67
web/scripts/build-locales.mjs
Normal file
67
web/scripts/build-locales.mjs
Normal file
@ -0,0 +1,67 @@
|
||||
import { spawnSync } from "child_process";
|
||||
import fs from "fs";
|
||||
import path from "path";
|
||||
import process from "process";
|
||||
|
||||
const localizeRules = JSON.parse(fs.readFileSync("./lit-localize.json", "utf-8"));
|
||||
|
||||
function compareXlfAndSrc(loc) {
|
||||
const xlf = path.join("./xliff", `${loc}.xlf`);
|
||||
const src = path.join("./src/locales", `${loc}.ts`);
|
||||
|
||||
// Returns false if: the expected XLF file doesn't exist, The expected
|
||||
// generated file doesn't exist, or the XLF file is newer (has a higher date)
|
||||
// than the generated file. The missing XLF file is important enough it
|
||||
// generates a unique error message and halts the build.
|
||||
|
||||
try {
|
||||
var xlfStat = fs.statSync(xlf);
|
||||
} catch (_error) {
|
||||
console.error(`lit-localize expected '${loc}.xlf', but XLF file is not present`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
try {
|
||||
var srcStat = fs.statSync(src);
|
||||
} catch (_error) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// if the xlf is newer (greater) than src, it's out of date.
|
||||
if (xlfStat.mtimeMs > srcStat.mtimeMs) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// For all the expected files, find out if any aren't up-to-date.
|
||||
|
||||
const upToDate = localizeRules.targetLocales.reduce(
|
||||
(acc, loc) => acc && compareXlfAndSrc(loc),
|
||||
true,
|
||||
);
|
||||
|
||||
if (!upToDate) {
|
||||
const status = spawnSync("npm", ["run", "build-locales:build"], { encoding: "utf8" });
|
||||
|
||||
// Count all the missing message warnings
|
||||
const counts = status.stderr.split("\n").reduce((acc, line) => {
|
||||
const match = /^([\w-]+) message/.exec(line);
|
||||
if (!match) {
|
||||
return acc;
|
||||
}
|
||||
acc.set(match[1], (acc.get(match[1]) || 0) + 1);
|
||||
return acc;
|
||||
}, new Map());
|
||||
|
||||
const locales = Array.from(counts.keys());
|
||||
locales.sort();
|
||||
|
||||
const report = locales
|
||||
.map((locale) => `Locale '${locale}' has ${counts.get(locale)} missing translations`)
|
||||
.join("\n");
|
||||
|
||||
console.log(`Translation tables rebuilt.\n${report}\n`);
|
||||
}
|
||||
|
||||
console.log("Locale ./src is up-to-date");
|
||||
@ -5,13 +5,12 @@ import { fileURLToPath } from "url";
|
||||
const __dirname = fileURLToPath(new URL(".", import.meta.url));
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
function* walkFilesystem(dir: string): Generator<string, undefined, any> {
|
||||
function* walkFilesystem(dir) {
|
||||
const openeddir = fs.opendirSync(dir);
|
||||
if (!openeddir) {
|
||||
return;
|
||||
}
|
||||
|
||||
let d: fs.Dirent | null;
|
||||
let d;
|
||||
while ((d = openeddir?.readSync())) {
|
||||
if (!d) {
|
||||
break;
|
||||
@ -24,13 +23,14 @@ function* walkFilesystem(dir: string): Generator<string, undefined, any> {
|
||||
}
|
||||
|
||||
const import_re = /^(import \w+ from .*\.css)";/;
|
||||
function extractImportLinesFromFile(path: string) {
|
||||
|
||||
function extractImportLinesFromFile(path) {
|
||||
const source = fs.readFileSync(path, { encoding: "utf8", flag: "r" });
|
||||
const lines = source?.split("\n") ?? [];
|
||||
return lines.filter((l) => import_re.test(l));
|
||||
}
|
||||
|
||||
function createOneImportLine(line: string) {
|
||||
function createOneImportLine(line) {
|
||||
const importMatch = import_re.exec(line);
|
||||
if (!importMatch) {
|
||||
throw new Error("How did an unmatchable line get here?");
|
||||
@ -43,15 +43,16 @@ function createOneImportLine(line: string) {
|
||||
}
|
||||
|
||||
const isSourceFile = /\.ts$/;
|
||||
|
||||
function getTheSourceFiles() {
|
||||
return Array.from(walkFilesystem(path.join(__dirname, "..", "src"))).filter((path) =>
|
||||
isSourceFile.test(path),
|
||||
);
|
||||
}
|
||||
|
||||
function getTheImportLines(importPaths: string[]) {
|
||||
const importLines: string[] = importPaths.reduce(
|
||||
(acc: string[], path) => [...acc, extractImportLinesFromFile(path)].flat(),
|
||||
function getTheImportLines(importPaths) {
|
||||
const importLines = importPaths.reduce(
|
||||
(acc, path) => [...acc, extractImportLinesFromFile(path)].flat(),
|
||||
[],
|
||||
);
|
||||
const uniqueImportLines = new Set(importLines);
|
||||
15
web/scripts/check-spelling.mjs
Normal file
15
web/scripts/check-spelling.mjs
Normal file
@ -0,0 +1,15 @@
|
||||
import { execSync } from "child_process";
|
||||
import path from "path";
|
||||
|
||||
const projectRoot = execSync("git rev-parse --show-toplevel", { encoding: "utf8" }).replace(
|
||||
"\n",
|
||||
"",
|
||||
);
|
||||
const cmd = [
|
||||
"codespell -D -",
|
||||
`-D ${path.join(projectRoot, ".github/codespell-dictionary.txt")}`,
|
||||
`-I ${path.join(projectRoot, ".github/codespell-words.txt")}`,
|
||||
"-S './src/locales/**' ./src -s",
|
||||
].join(" ");
|
||||
|
||||
console.log(execSync(cmd, { encoding: "utf8" }));
|
||||
76
web/scripts/eslint-precommit.mjs
Normal file
76
web/scripts/eslint-precommit.mjs
Normal file
@ -0,0 +1,76 @@
|
||||
import { execFileSync } from "child_process";
|
||||
import { ESLint } from "eslint";
|
||||
import path from "path";
|
||||
import process from "process";
|
||||
|
||||
// Code assumes this script is in the './web/scripts' folder.
|
||||
const projectRoot = execFileSync("git", ["rev-parse", "--show-toplevel"], {
|
||||
encoding: "utf8",
|
||||
}).replace("\n", "");
|
||||
process.chdir(path.join(projectRoot, "./web"));
|
||||
|
||||
const eslintConfig = {
|
||||
overrideConfig: {
|
||||
env: {
|
||||
browser: true,
|
||||
es2021: true,
|
||||
},
|
||||
extends: [
|
||||
"eslint:recommended",
|
||||
"plugin:@typescript-eslint/recommended",
|
||||
"plugin:lit/recommended",
|
||||
"plugin:custom-elements/recommended",
|
||||
"plugin:storybook/recommended",
|
||||
"plugin:sonarjs/recommended",
|
||||
],
|
||||
parser: "@typescript-eslint/parser",
|
||||
parserOptions: {
|
||||
ecmaVersion: 12,
|
||||
sourceType: "module",
|
||||
},
|
||||
plugins: ["@typescript-eslint", "lit", "custom-elements", "sonarjs"],
|
||||
rules: {
|
||||
"indent": "off",
|
||||
"linebreak-style": ["error", "unix"],
|
||||
"quotes": ["error", "double", { avoidEscape: true }],
|
||||
"semi": ["error", "always"],
|
||||
"@typescript-eslint/ban-ts-comment": "off",
|
||||
"sonarjs/cognitive-complexity": ["error", 9],
|
||||
"sonarjs/no-duplicate-string": "off",
|
||||
"sonarjs/no-nested-template-literals": "off",
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const porcelainV1 = /^(..)\s+(.*$)/;
|
||||
const gitStatus = execFileSync("git", ["status", "--porcelain", "."], { encoding: "utf8" });
|
||||
|
||||
const statuses = gitStatus.split("\n").reduce((acc, line) => {
|
||||
const match = porcelainV1.exec(line.replace("\n"));
|
||||
if (!match) {
|
||||
return acc;
|
||||
}
|
||||
const [status, path] = Array.from(match).slice(1, 3);
|
||||
return [...acc, [status, path.split("\x00")[0]]];
|
||||
}, []);
|
||||
|
||||
const isModified = /^(M|\?|\s)(M|\?|\s)/;
|
||||
const modified = (s) => isModified.test(s);
|
||||
|
||||
const isCheckable = /\.(ts|js|mjs)$/;
|
||||
const checkable = (s) => isCheckable.test(s);
|
||||
|
||||
const updated = statuses.reduce(
|
||||
(acc, [status, filename]) =>
|
||||
modified(status) && checkable(filename) ? [...acc, path.join(projectRoot, filename)] : acc,
|
||||
[],
|
||||
);
|
||||
|
||||
const eslint = new ESLint(eslintConfig);
|
||||
const results = await eslint.lintFiles(updated);
|
||||
const formatter = await eslint.loadFormatter("stylish");
|
||||
const resultText = formatter.format(results);
|
||||
const errors = results.reduce((acc, result) => acc + result.errorCount, 0);
|
||||
|
||||
console.log(resultText);
|
||||
process.exit(errors > 1 ? 1 : 0);
|
||||
@ -4,16 +4,12 @@ import pseudolocale from "pseudolocale";
|
||||
import { fileURLToPath } from "url";
|
||||
|
||||
import { makeFormatter } from "@lit/localize-tools/lib/formatters/index.js";
|
||||
import type { Message, ProgramMessage } from "@lit/localize-tools/lib/messages.d.ts";
|
||||
import { sortProgramMessages } from "@lit/localize-tools/lib/messages.js";
|
||||
import { TransformLitLocalizer } from "@lit/localize-tools/lib/modes/transform.js";
|
||||
import type { Config } from "@lit/localize-tools/lib/types/config.d.ts";
|
||||
import type { Locale } from "@lit/localize-tools/lib/types/locale.d.ts";
|
||||
import type { TransformOutputConfig } from "@lit/localize-tools/lib/types/modes.d.ts";
|
||||
|
||||
const __dirname = fileURLToPath(new URL(".", import.meta.url));
|
||||
const pseudoLocale: Locale = "pseudo-LOCALE" as Locale;
|
||||
const targetLocales: Locale[] = [pseudoLocale];
|
||||
const pseudoLocale = "pseudo-LOCALE";
|
||||
const targetLocales = [pseudoLocale];
|
||||
const baseConfig = JSON.parse(readFileSync(path.join(__dirname, "../lit-localize.json"), "utf-8"));
|
||||
|
||||
// Need to make some internal specifications to satisfy the transformer. It doesn't actually matter
|
||||
@ -21,27 +17,28 @@ const baseConfig = JSON.parse(readFileSync(path.join(__dirname, "../lit-localize
|
||||
// is in their common parent class, but I had to pick one. Everything else here is just pure
|
||||
// exploitation of the lit/localize-tools internals.
|
||||
|
||||
const config: Config = {
|
||||
const config = {
|
||||
...baseConfig,
|
||||
baseDir: path.join(__dirname, ".."),
|
||||
targetLocales,
|
||||
output: {
|
||||
...baseConfig,
|
||||
...baseConfig.output,
|
||||
mode: "transform",
|
||||
},
|
||||
resolve: (path: string) => path,
|
||||
} as Config;
|
||||
resolve: (path) => path,
|
||||
};
|
||||
|
||||
const pseudoMessagify = (message: ProgramMessage) => ({
|
||||
const pseudoMessagify = (message) => ({
|
||||
name: message.name,
|
||||
contents: message.contents.map((content) =>
|
||||
typeof content === "string" ? pseudolocale(content, { prepend: "", append: "" }) : content,
|
||||
),
|
||||
});
|
||||
|
||||
const localizer = new TransformLitLocalizer(config as Config & { output: TransformOutputConfig });
|
||||
const { messages } = localizer.extractSourceMessages();
|
||||
const localizer = new TransformLitLocalizer(config);
|
||||
const messages = localizer.extractSourceMessages().messages;
|
||||
const translations = messages.map(pseudoMessagify);
|
||||
const sorted = sortProgramMessages([...messages]);
|
||||
const formatter = makeFormatter(config);
|
||||
formatter.writeOutput(sorted, new Map<Locale, Message[]>([[pseudoLocale, translations]]));
|
||||
|
||||
formatter.writeOutput(sorted, new Map([[pseudoLocale, translations]]));
|
||||
Reference in New Issue
Block a user