Dart build_runner can only scan/read/write files in the web/ directory?

Question

I am developing a Flutter application, and would like to write a build script to converts some kind of raw files (in CSV) to formatted JSON files to be included as Flutter assets.

By using libraries like json_serializable and jaguar_serializer I learned about build_runner, so it looks me that writing my own Builder and invoke it via build_runner is a sensible way.

As there are very limited resources about writing our own build script, I started by modifying the examples found here. But I got stuck when trying to change the path of the input and output file: when I run flutter pub pub run build_runner build, it turned out that Dart only searches the matching files in the [project_dir]/web directory, and only allows me to write files to this web directory. So this code

buildStep.writeAsString(new AssetId(buildStep.inputId.package, 'assets/resources/foo.json'), '[]');

will produce the following exception:

UnexpectedOutputException: myapp|assets/resources/foo.json
Expected only: {myapp|web/.json}
[SEVERE] Failed after 24.4s

The fact that json_serializable and jaguar_serializer have the freedom to generate code anywhere seems to indicate that this is something wrong with my configuration. But I can't found this web thing anywhere in my code nor the build.yaml file, so this really puzzling.

FWIW, here's the content of my builder.yaml file:

builders:
  jsonBuilder:
    import: "package:myapp/builder.dart"
    builder_factories: ["jsonFileBuilder"]
    build_extensions: {"source.csv": [".json"]}
    build_to: source
    auto_apply: root_package

And here's the builder.dart file

import 'dart:async';
import 'package:build/build.dart';

Builder jsonFileBuilder(BuilderOptions options) => new JsonFileBuilder();

class JsonFileBuilder implements Builder {
  @override
  Future build(BuildStep buildStep) async {
    await buildStep.writeAsString(
        new AssetId(buildStep.inputId.package, 'assets/resources/foo.json'),
        'hello');
  }

  @override
  final buildExtensions = const {
    'source.csv': const ['.json']
  };
}

Thanks in advance!

Answer 1

Well, it appears that the question is actually twofold.

About the input

My problem was that my build script won't run due to no matching files found if I put the source CSV file under some arbitrary directory (like offline). But when I put the file under web (as in the example) the script gets called. This gave me an illusion that file scanning is limited to web/. The fact is, there is a list of hard-coded file/directory whitelist:

const List<String> _defaultRootPackageWhitelist = const [
  'benchmark/**',
  'bin/**',
  'example/**',
  'lib/**',
  'test/**',
  'tool/**',
  'web/**',
  'pubspec.yaml',
  'pubspec.lock',
];

My build script also get called if I put the CSV file under lib/.

To have my non-whitelisted file being appended to the list, I have to add the following portions to build.yaml.

targets:
  $default:
    sources: 
    # Need to replicate the default whitelist here or other build will break:
    - "benchmark/**"
    - "bin/**"
    - "example/**"
    - "lib/**"
    - "test/**"
    - "tool/**"
    - "web/**"
    - "pubspec.yaml"
    - "pubspec.lock"
    # My new source
    - "offline/source.csv"

About the output

My problem was that I cannot write to arbitrary directory, the build system keeps complaining that I can only write to web/. So I thought that file writing was locked to web/. Again, this is not entirely true, because file writing is actually locked to the directory of the input file. This means that I could only write to offline/ after I successfuly have build runner recognize my file in this directory.

The checking of file writing permission is controlled by this method in build_step_impl.dart:

  void _checkOutput(AssetId id) {
    if (!_expectedOutputs.contains(id)) {
      throw new UnexpectedOutputException(id, expected: _expectedOutputs);
    }
  }

This _expectedOutputs is the flattened value of the buildExtensions map of my builder class. So if I really want to write to assets/resources/foo.json I have to write this:

  @override
  final buildExtensions = const {
    'source.csv': const ['/../../assets/resources/foo.json']
  };

Similar adjustments to build.yaml have to be made:

build_extensions: {".csv": ["/../../assets/resources/foo.json"]}

This basically solves most of my problems, but some questions remain:

First, I was planning to write a build script that reads a single file and writes to multiple files. The number of files and their filenames depend on the content of the source file. Given the checking code, this doesn't seem to be a supported feature. Not a blocker, but somewhat inconvenient.

The second one is about the output file path. You can see that my source file reisdes in [projectDir]/offline and I want to write to [projectDir]/assets, but the output filename reads /../../assets/..., which is two levels up. The is because the original filename (without extension) is prepended to the output path, which is why one would see .g.dart in builder configuration that generates Dart code. I need to write /../../assets so the full output path of my file would become [projectDir]/offline/source/../../assets to make it resolve to the desired output file path. But the default behavior gives me an impression that the build system doesn't expect my script to write to any place out of the directory of the current file, and what I am doing is a hack or an abuse to the system.