What is the fastest route to regenerate Types in a Program instance after *some* files modified?

Question

I have a Program instance in which I've modified a few nodes in a few SourceFile. I'd like to regenerate the types for the modified SourceFiles in the most performant way possible. (Note: I am not emitting)

The initial way I've done this is:

• Create a CompilerHost and hook getSourceFile to allow it to serve modified SourceFile if available
• Run a transformer on all SourceFile, update modified collection from result
• Recreate Program using the same compilerHost (serves updated files)

I'm trying to determine if there is a more optimal way to do this.

I've investigated the possibility of using a LanguageService, but it appears that ts.createLanguageService simply triggers a re-creation of Program when files are changed.

Another route I've looked at is ts.createWatchProgram. It seems to use similar logic in implementing an isProgramUptoDate() function which runs createProgram if filenames have changed. For my purposes, it's looking like both would add unnecessary complexity and presumably drag over my original route.

However, because I don't have much experience with either yet, it's possible that I'm missing something.

Answer 1

After taking the day to do some heavy perf testing and digging I've learned a few things.

It seems the best results can be gained through caching unchanged SourceFile and hooking CompilerHost to return from cache, when available. (I was recently told that the LanguageService works in a similar way)

If you're only modifying a few files, this can have a tremendous boost in performance. When I modified none of the 230 TS files, Program reloaded in 1ms consistently without supplying oldProgram and 100ms when supplying it.

However, even with many files modified, gains can be still be realized.

The big take-away, for me, is that each new SourceFile which gets served from the CompilerHost will have to have its types walked again, which can get expensive.

So the ways to improve speed are the following:

1. Create a CompilerHost and hook getSourceFile() to serve the SourceFile from a cache map, when available.

2. If you can get away with not using the TypeChecker during ts.transform(), don't create a Program instance until after you transform. (use your compilerHost to load the SourceFile[] for use with ts.transform())

3. During transform, keep track of which files actually get modified and only update those files in your cache.

Example code:

import * as ts from 'typescript';
import * as glob from 'glob';
import {
  CompilerHost, CompilerOptions, HeritageClause, IndexedAccessTypeNode, SourceFile, SyntaxKind, TypeReferenceNode
} from 'typescript';


/* ********************************************************* *
 * Helpers
 * ********************************************************* */

export const nodeIsKind = <T extends ts.Node = never>(node: ts.Node, ...kind: ts.SyntaxKind[]): node is T =>
  kind.some(k => node.kind === k);


/* ********************************************************* *
 * Compiler
 * ********************************************************* */

function createHookedCompilerHost(hostFiles: Map<string, SourceFile>, compilerOptions: CompilerOptions) {
  const host = ts.createCompilerHost(compilerOptions);
  const originalGetSourceFile = host.getSourceFile as Function;

  return Object.assign(host, {
    getSourceFile(fileName: string, languageVersion: ts.ScriptTarget) {
      return hostFiles.get(fileName) || originalGetSourceFile(...arguments);
    }
  });
}

function transformNodes(program: ts.Program) {
  const srcFiles = program.getSourceFiles() as SourceFile[];
  const updatedFiles = new Set<string>();

  const transformer = (context: ts.TransformationContext) => {
    function visit(fileName: string) {
      return (node: ts.Node): ts.Node => {
        /* Ignore these */
        if (nodeIsKind<HeritageClause>(node, SyntaxKind.HeritageClause)) return node;

        /* Wrap in tuple */
        if (nodeIsKind<TypeReferenceNode>(node, SyntaxKind.TypeReference) || nodeIsKind<IndexedAccessTypeNode>(node, SyntaxKind.IndexedAccessType)) {
          updatedFiles.add(fileName); // Mark file as modified
          return ts.createTupleTypeNode([ node ]);
        }

        return ts.visitEachChild(node, visit(fileName), context);
      }
    }

    return (sourceFile: ts.SourceFile) => ts.visitNode(sourceFile, visit(sourceFile.fileName));
  };

  const { transformed } = ts.transform(srcFiles, [ transformer ], program.getCompilerOptions());
  return transformed.filter(sourceFile => updatedFiles.has(sourceFile.fileName));
}


/* ********************************************************* *
 * Config
 * ********************************************************* */

const fileNames = glob.sync('./test/assets/**/*.ts');

const compilerOptions = {
  noEmit: true,
  target: ts.ScriptTarget.ES5,
  module: ts.ModuleKind.CommonJS,
  strictNullChecks: false,
};


/* ********************************************************* *
 * Main
 * ********************************************************* */

/* Setup Host & Program */
const hostFiles = new Map<string, SourceFile>();
const host = createHookedCompilerHost(hostFiles, compilerOptions);

/* Load SourceFiles */
const sourceFiles = fileNames.map(fileName => host.getSourceFile(fileName, compilerOptions.target));

/* Build Program */
let program = ts.createProgram(fileNames, compilerOptions, host);

/* Pre-cache sourceFiles */
program.getSourceFiles().forEach(srcFile => hostFiles.set(srcFile.fileName, srcFile));

/* Transform files & update affected SourceFiles */
const transformed = transformNodes(program);
for (const sourceFile of transformed)
  hostFiles.set(
    sourceFile.fileName,
    ts.createSourceFile(sourceFile.fileName, ts.createPrinter().printFile(sourceFile), sourceFile.languageVersion)
  );

/* Re-generate Program */
program = ts.createProgram(fileNames, compilerOptions, host);

/* Do what you need with the TypeChecker here */

Feel free to leave a comment if you have questions or need help.