StackExchange.Precompilation - How can I unit test precompilation diagnostics?

Question

Background

I'm using StackExchange.Precompilation to implement aspect-oriented programming in C#. See my repository on GitHub.

The basic idea is that client code will be able to place custom attributes on members, and the precompiler will perform syntax transformations on any members with those attributes. A simple example is the NonNullAttribute I created. When NonNullAttribute is placed on a parameter p, the precompiler will insert

if (Object.Equals(p, null)) throw new ArgumentNullException(nameof(p));

at the beginning of the method body.

---

Diagnostics are awesome...

I would like to make it difficult to use these attributes incorrectly. The best way I have found (aside from intuitive design) is to create compile-time Diagnostics for invalid or illogical uses of attributes.

For example, NonNullAttribute does not make sense to use on value-typed members. (Even for nullable value-types, because if you wanted to guarantee they weren't null then a non-nullable type should be used instead.) Creating a Diagnostic is a great way to inform the user of this error, without crashing the build like an exception.

---

...but how do I test them?

Diagnostics are a great way to highlight errors, but I also want to make sure my diagnostic creating code does not have errors. I would like to be able to set up a unit test that can precompile a code sample like this

public class TestClass {
    public void ShouldCreateDiagnostic([NonNull] int n) { }       
}

and confirm that the correct diagnostic is created (or in some cases that no diagnostics have been created).

Can anyone familiar with StackExchange.Precompilation give me some guidance on this?

---

Solution:

The answer given by @m0sa was incredibly helpful. There are a lot of details to the implementation, so here is the unit test actually looks like (using NUnit 3). Note the using static for SyntaxFactory, this removes a lot of clutter in the syntax tree construction.

using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using NUnit.Framework;
using StackExchange.Precompilation;
using static Microsoft.CodeAnalysis.CSharp.SyntaxFactory;

namespace MyPrecompiler.Tests {

    [TestFixture]
    public class NonNull_CompilationDiagnosticsTest {

        [Test]
        public void NonNullAttribute_CreatesDiagnosticIfAppliedToValueTypeParameter() {

            var context = new BeforeCompileContext {
                Compilation = TestCompilation_NonNullOnValueTypeParameter(),
                Diagnostics = new List<Diagnostic>()
            };

            ICompileModule module = new MyPrecompiler.MyModule();

            module.BeforeCompile(context);

            var diagnostic = context.Diagnostics.SingleOrDefault();

            Assert.NotNull(diagnostic);
            Assert.AreEqual("MyPrecompiler: Invalid attribute usage", 
                            diagnostic.Descriptor.Title.ToString()); //Must use ToString() because Title is a LocalizeableString
        }

        //Make sure there are spaces before the member name, parameter names, and parameter types.
        private CSharpCompilation TestCompilation_NonNullOnValueTypeParameter() {
            return CreateCompilation(
                MethodDeclaration(ParseTypeName("void"), Identifier(" TestMethod"))
                .AddParameterListParameters(
                    Parameter(Identifier(" param1"))
                        .WithType(ParseTypeName(" int"))
                        .AddAttributeLists(AttributeList()
                                            .AddAttributes(Attribute(ParseName("NonNull"))))));
        }

        //Make sure to include Using directives
        private CSharpCompilation CreateCompilation(params MemberDeclarationSyntax[] members) {

            return CSharpCompilation.Create("TestAssembly")
               .AddReferences(References)
               .AddSyntaxTrees(CSharpSyntaxTree.Create(CompilationUnit()
                    .AddUsings(UsingDirective(ParseName(" Traction")))
                    .AddMembers(ClassDeclaration(Identifier(" TestClass"))
                                .AddMembers(members))));
        }

        private string runtimePath = @"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\";

        private MetadataReference[] References =>
            new[] {
                MetadataReference.CreateFromFile(runtimePath + "mscorlib.dll"),
                MetadataReference.CreateFromFile(runtimePath + "System.dll"),
                MetadataReference.CreateFromFile(runtimePath + "System.Core.dll"),
                MetadataReference.CreateFromFile(typeof(NonNullAttribute).Assembly.Location)
            };
    }
}

Answer 1

I figure you want to add you diagnostics before the actual emit / compilation, so the steps would be:

1. create your CSharpCompilation, make sure it has no diagnostic errors before going further
2. create an BeforeCompileContext, and populate it with the compilation and an empty List<Diagnostic>
3. create an instance of your ICompileModule and call ICompileModule.BeforeCompile with the context from step 2
4. check that it contains the required Diagnostic