Garbage Collector's sabotage of my message pump

I spent my last days with a boring problem. I implemented a Windows service listening to WM_DEVICECHANGE messages (which will be sent par example, when you plug-in a new USB device). For this I implemented a NativeWindow, whose only purpose was to implement the window procedure for some custom message processing:

internal sealed class DeviceChangeHandler : NativeWindow
{
  private const int WM_DEVICECHANGE = 0x219;
  private const int DBT_DEVNODES_CHANGED = 0x7;

  public DeviceChangeHandler()
  {
    CreateHandle(new CreateParams());
  }

  protected override void WndProc(ref Message msg)
  {
    if (msg.Msg == WM_DEVICECHANGE && 
        msg.WParam.ToInt32() == DBT_DEVNODES_CHANGED)
    {
      // device change detected
      ...
    }

    base.WndProc(ref msg);
  }
}

In my service itself, I started a new Thread with the following ThreadStart:

private void RunDeviceChangeHandler()
{
  // create window
  DeviceChangeHandler deviceChangeHandler = new DeviceChangeHandler();

  // run message pump
  Application.Run();
}  

When I debugged the code, it worked always. Also my release build worked - but unfortunately not always. Sometimes I got the messages, sometimes not. Testing was tedious, since Windows takes some time until sending the message, especially after disconnecting the device.

Finally I remembered a tool I used 10 years ago: Spy++, which is still part of Visual Studio. I saw that, when I didn't get messages, there was also no window. That explained to me, why I didn't get the messages. But the question was now: why was there no window?

Next I defined a caption for my window (via the CreateParams). And I implemented a loop in which I call every 5 seconds FindWindow with the just specified caption. Now I saw that FindWindow was able to find my window. But only for some iterations, sooner or later it returned only 0.

By chance, I added tracing to the finalizer of my NativeWindow implementation. Now I saw that the finalizer was called some time, and afterwards FindWindow returned only 0. My first idea was that there was an exception in Application.Run(). But this was to easy.

The problem is optimization done by the just-in-time compiler. Since the variable deviceChangeHandler is no longer referenced after Application.Run(), it is a candidate for garbage collection. This I could verify by a call of GC.Collect(). Afterwards the windows was always gone.

The solution is to make the variable ineligible for garbage collection. For this purpose exists the method GC.KeepAlive().
With my modified ThreadStart, everything worked as expected:

private void RunDeviceChangeHandler()
{
  // create window
  DeviceChangeHandler deviceChangeHandler = new DeviceChangeHandler();

  // run message pump
  Application.Run();

  // make deviceChangeHandler ineligible for garbage collection
  GC.KeepAlive(deviceChangeHandler);
}  

 

As with the most complex problems, the solution is only one line. The art is to insert it at the right place.

Pitfall with ASP.NET Web Api, OWIN and FxCop

As I wrote in Self-host ASP.NET Web API and SignalR together, you can easily combine ASP.NET Web Api and OWIN. My Startup class contained at least the Configuration method:

public void Configuration(IAppBuilder app)
{
  HttpConfiguration config = new HttpConfiguration();
  config.Routes.MapHttpRoute("API Default", "api/{controller}/{id}", new { id = RouteParameter.Optional });
  app.UseWebApi(config);
}

With this, everything worked fine.

(Unfortunately) I am a well-behaved man. Therefore I ran the code analysis (aka FxCop). It returned the warning

CA2000: Microsoft.Reliability: In method 'Startup.Configuration(IAppBuilder)', call System.IDisposable.Dispose on object 'config' before all references to it are out of scope.

Understandable for me. And since Microsoft suggests to fix it, I changed my method to:

public void Configuration(IAppBuilder app)
{
  using (HttpConfiguration config = new HttpConfiguration())
  {
    config.Routes.MapHttpRoute("API Default", "api/{controller}/{id}", new { id = RouteParameter.Optional });
    app.UseWebApi(config);
  }
}

Sadly, I didn’t test it immediately. Instead I fixed a lot of other FxCop warnings. And I improved my coding otherwise. Finally I forgot this special fix.

When I finally put all together, I got only HTTP 500 Internal Server Error, without any further information. A lot of people suggest in such a case to debug the server, but also there no exception was thrown. It simply didn’t work. I slimmed down my coding until I removed the using statement. And then it worked again!

So it’s better to keep the original code and to add only a SuppressMessage attribute:

[SuppressMessage("Microsoft.Reliability", "CA2000:Dispose objects before losing scope",
                 Justification = "HttpConfiguration must not be disposed, otherwise web api will not work")]
public void Configuration(IAppBuilder app)
{
  HttpConfiguration config = new HttpConfiguration();
  config.Routes.MapHttpRoute("API Default", "api/{controller}/{id}", new { id = RouteParameter.Optional });
  app.UseWebApi(config);
}

Claims-based authentication in a web application using ACS

I like the idea to use existing user accounts for authentication instead of managing users by myself.
Otherwise I have to store user names and passwords in my database. Therefore I have to provide a user interface to create, change or delete them. And most important, I have to ensure that everything is secure. Nothing I really want to bother when I am developing some cool application.
Also for the user does not like to create a new account for every web site she visits. This is quite boring, especially as she should use different passwords for each site. That arises the need to store somewhere the different passwords?
To cut a long story short: why not use existing user accounts like Google, Windows Live ID, Facebook, … ?

Integrating ACS

A good starter is How to: Create My First Claims-Aware ASP.NET Application Using ACS. Unfortunately, this article is a little bit outdated in between (written in April 2011). Mainly the toolset (Visual Studio 2012, Identity and Access Tool) changed.

Web Application

To demonstrate the principle, start with an ASP.NET Empty Web Application. Add a Web Form with some text in the body like

Hello <%= User.Identity.Name %>

When you start the debugger, you should see a web page with text Hello – but without user name, since no authentication is done.

Configure ACS

Before we can add authentication, we have to do some configuration. Just follow the first steps from the document mentioned above (How to: Create My First Claims-Aware ASP.NET Application Using ACS):

• Step 1 - Create an Access Control Namespace
• Step 2 – Launch the ACS Management Portal
• Step 3 – Add Identity Providers

Identity and Access Tool

With Visual Studio 2012 the ACS integration is no longer done with a STS Reference. Instead, the Identity and Access Tool has to be used. You can download it from Visual Studio Gallery. After installing it, you can start it in the Solution Explorer in the project’s context menu.
Since we want to connect to ACS, select the third option Use the Windows Azure Access Control Service:

Now we have to configure the providers. Just click on the Configure link in the middle:

The ACS namespace is easy to know. It’s just the name you chose. The management key is not so easy to find:

• Open Windows Azure Portal
• Manage your ACS namespace
• Select Management service
• Click ManagementClient
• Click Symmetric Key

Here you can copy the key:

The next step is to select the providers you want to use in your application. And to specify realm and return url of your application. By default the are initialized with the url used for debugging in Visual Studio.
After clicking OK, you get a lot of stuff generated in your Web.config. Additionally, you can find in the Azure Portal an additional Relying Party Application (originally, this was step 4 from How to: Create My First Claims-Aware ASP.NET Application Using ACS.
When you start now the debugger, the first page is something like

The user can select with which identity provider she wants to use. After the login, the start page of your application should be displayed, showing the user name.

Fiddler

When you check the logon process in Fiddler, you will see the following requests:

• http://localhost:58235/
  The URL of the application itself, redirects to
• https://markus.accesscontrol.windows.net/v2/wsfederation?...
  The page to select the identity provider; selecting one forwards to
• https://accounts.google.com/o/openid2/auth?...
  The login itself, here at Google; after confirmation redirects back to
• https://markus.accesscontrol.windows.net/v2/openid?...
  ACS redirects back to the application itself
• http://localhost:58235/

Claims

Since .NET 4.5 every Principal is based on a ClaimsPrincipal. That means every attribute of the user is a claim. It’s easy to query or display them:

System.Security.Claims.ClaimsPrincipal cp = (System.Security.Claims.ClaimsPrincipal)User;
foreach (var claim in cp.Claims)
{
}

With a Goggle account par example, you have at least for claims:

Specific claims

You can rely only on the existence of the claims http://schemas.microsoft.com/accesscontrolservice/2010/07/claims/identityprovider and http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier. All other claims are optional, also such convenient things like http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name or http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress. Par example Windows Live ID does not provide this information due to security restrictions.
This is especially boring, since the name claim will be mapped to User.Identity.Name. When the claim is missing, the Name property is null. So maybe you get runtime errors because of that. Therefore it could be a good idea to provide a default value for this claim.
You can do it in the Windows Azure Portal in the section Rule groups. Select the rule group of your application, and then you should see Passthrough rules, which forward the claims from the identity provider to your application. Here you add your own rules, e.g.:

• Identity Provider: Windows Live ID
• Input claim type: Any
• Input claim value: Any
• Output claim type: http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name
• Output claim value: ???
• Description: Default name for Windows LiveID

With this rule in place, every user authenticated with Windows Live ID has the name ???.

Roles

Also roles are now claims. That means you can also define a rule to apply a role:

• Identity Provider: Windows Live ID
• Input claim type: http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier
• Input claim value: (the name identifier)
• Output claim type: http://schemas.microsoft.com/ws/2008/06/identity/claims/role
• Output claim value: admin
• Description: Admin role for xxx

Now you can check the existence of the role, e.g. (or 1 one of the other 1,000,000 possibilities):

User.IsInRole("admin")

ClaimsAuthenticationManager

Adding roles or names via rules is not very feasible when you have more than 2 or 3 users. Therefore it is better to implement a ClaimsAuthenticationManager to extend the claims processing pipeline. Here you can modify the claims as you want:

public class MyClaimsAuthenticationManager : ClaimsAuthenticationManager
{
  public override ClaimsPrincipal Authenticate(string resourceName, ClaimsPrincipal incomingPrincipal)
  {
    if (incomingPrincipal != null && incomingPrincipal.Identity.IsAuthenticated)
    {
      ClaimsIdentity claimsIdentity = (ClaimsIdentity)incomingPrincipal.Identity;
      string identityProvider =
        claimsIdentity.Claims
          .Where(c => c.Type == "http://schemas.microsoft.com/accesscontrolservice/2010/07/claims/identityprovider")
          .Select(c => c.Value)
          .FirstOrDefault();
      string nameIdentifier = 
        claimsIdentity.Claims
          .Where(c => c.Type ==ClaimTypes.NameIdentifier)
          .Select(c => c.Value)
          .FirstOrDefault();

      if (identityProvider == "uri:WindowsLiveID" && nameIdentifier == "FVUzvNwYGuC5cG4VYdWArf81SRj0QISjQpUIhaHonNE=")
      {
        claimsIdentity.AddClaim(new Claim(ClaimTypes.Name, "Markus Wagner"));
        claimsIdentity.AddClaim(new Claim(ClaimTypes.Role, "admin"));
      }
    }

    return incomingPrincipal;
  }
}

Finally you have to configure your application to use the new ClaimsAuthenticationManager. This can be done in the Web.config:

<system.identityModel>
  <identityConfiguration>
    <claimsAuthenticationManager type="AcsAuthentication.MyClaimsAuthenticationManager, AcsAuthentication" />
  </identityConfiguration>
</system.identityModel>

For sure, in a real world application, you will not hard-code the claims here. Instead you will take them from a configuration file or a database. But the principle will stay the same.

Conclusion

For authentication, ACS is already a good alternative. Especially since you do not have to manage passwords.
With authentication it gets more complicated. Some prerequisites exist, but it isn't really comfortable.

Writing your own Glimpse Plugin

As I wrote in my last post, Glimpse is a great tool to diagnose your web application. And there is already a lot of plugins available. But maybe you have the need to get some additional info, for which no plugin is available so far. In such a case it is quite easy to write your own plugin.
I will show it with a plugin for Log4Net. Before you ask, yes there is already a Glimpse.Log4Net plugin. Unfortunately this package is outdated. It was written for Glimpse 0.86, but with 1.x it doesn't work anymore.

UPDATE 15-Jun-2013: In between the Glimpse.Log4Net plugin has been updated to support Glimpse 1.x. But nevertheless the how to below should be interesting enough, since there is not a lot of documentation out there.

Log4Net and Trace

It is not absolute necessary to use a special Log4Net plugin. Instead it is possible to redirect the Log4Net output to the Tracing infrastructure. And for this, Glimpse provides already a tab.
All you need is a new appender:

<appender name="TraceAppender" type="log4net.Appender.TraceAppender">
  <category value="%level" /> 
  <layout type="log4net.Layout.PatternLayout"> 
    <conversionPattern value="%logger - %message" /> 
  </layout> 
</appender>

After connecting it to a logger, you will see all relevant log lines in the trace tab. With the specification of the category tag, the trace output will have the same level as the original Log4Net log line. Therefore some of the outputs are prefixed with the corresponding icons:

 

Log4Net Plugin

The implementation of the plugin is straightforward: first we need a Log4Net appender, which collects the messages to display in Glimpse. And we need an implementation for the tab in the Glimpse window. The rest in some infrastructure around it.
So let's have a look at the details.

Log4NetMessage

This is a simple container for the data transferred from the Log4Net appender to Glimpse. The only prerequisite is, it has to implement the interface Glimpse.Core.Message.IMessage:

public class Log4NetMessage : IMessage
{
  public Log4NetMessage()
  {
    Id = Guid.NewGuid();
  }

  public Guid Id { get; private set; }
  public TimeSpan FromFirst { get; set; }
  public TimeSpan FromLast { get; set; }
  public string ThreadName { get; set; }
  public string Level { get; set; }
  public string LoggerName { get; set; }
  public string Message { get; set; }
}

Log4NetInspector

Next we need some additional infrastructure. The appender sends the message to the tab via a Glimpse.Core.Extensibility.IMessageBroker. This must be set from outside.
Therefore I added an implementation of Glimpse.Core.Extensibility.IInspector, whose Setup method will be called at startup. It takes the needed properties from the Glimpse context and stores them in static properties of my appender:

public class Log4NetInspector : IInspector
{
  public void Setup(IInspectorContext context)
  {
    GlimpseAppender.Initialize(context.MessageBroker, context.TimerStrategy);
  }
}

GlimpseAppender

Now it's time for the real stuff, first the appender. The implementation is quite easy, a bigger part of the code is about the calculation of the elapsed time since the last call (this part could be also skipped).

public class GlimpseAppender : AppenderSkeleton
{
  private static IMessageBroker _messageBroker;
  private static Func<IExecutionTimer> _timerStrategy;

  private static Stopwatch fromLastWatch;

  public static void Initialize(IMessageBroker messageBroker, Func<IExecutionTimer> timerStrategy)
  {
    _messageBroker = messageBroker;
    _timerStrategy = timerStrategy;
  }

  protected override void Append(LoggingEvent loggingEvent)
  {
    if (_timerStrategy != null && _messageBroker != null)
    {
      IExecutionTimer timer = _timerStrategy();
      if (timer != null)
      {
        _messageBroker.Publish(new Log4NetMessage
          {
            ThreadName = loggingEvent.ThreadName,
            Level = loggingEvent.Level.DisplayName,
            LoggerName = loggingEvent.LoggerName,
            Message = loggingEvent.RenderedMessage,
            FromFirst = timer.Point().Offset,
            FromLast = CalculateFromLast(timer)
          });
      }
    }
  }

  private static TimeSpan CalculateFromLast(IExecutionTimer timer)
  {
    if (fromLastWatch == null)
    {
      fromLastWatch = Stopwatch.StartNew();
      return TimeSpan.FromMilliseconds(0);
    }

    // Timer started before this request, reset it
    if (DateTime.Now - fromLastWatch.Elapsed < timer.RequestStart)
    {
      fromLastWatch = Stopwatch.StartNew();
      return TimeSpan.FromMilliseconds(0);
    }

    var result = fromLastWatch.Elapsed;
    fromLastWatch = Stopwatch.StartNew();
    return result;
  }
}

Log4NetTab

Now we are ready for the tab's implementation. It inherits from Glimpse.Core.Extensibility.TabBase, and implements the interfaces Glimpse.Core.Extensibility.ITabSetup, Glimpse.Core.Extensibility.IKey and Glimpse.Core.Extensibility.ITabLayout. As you see, the biggest part is the definition of the layout, how the data will be displayed on the tab.

public class Log4NetTab : TabBase, ITabSetup, IKey, ITabLayout
{
  private static readonly object layout = TabLayout.Create()
    .Row(r =>
    {
      r.Cell(0).WidthInPixels(80);
      r.Cell(1).WidthInPixels(80);
      r.Cell(2);
      r.Cell(3);
      r.Cell(4).WidthInPercent(15).Suffix(" ms").AlignRight().Prefix("T+ ").Class("mono");
      r.Cell(5).WidthInPercent(15).Suffix(" ms").AlignRight().Class("mono");
    }).Build();

  public override object GetData(ITabContext context)
  {
    return context.GetMessages<Log4NetMessage>().ToList();
  }

  public override string Name
  {
    get { return "Log4Net"; }
  }

  public void Setup(ITabSetupContext context)
  {
    context.PersistMessages<Log4NetMessage>();
  }

  public string Key
  {
    get { return "glimpse_log4net"; }
  }

  public object GetLayout()
  {
    return layout;
  }
}

Log4NetMessagesConverter

The final part is a converter, which knows how to display a Log4NetMessage in the layout of Log4NetTab:

public class Log4NetMessagesConverter : SerializationConverter<IEnumerable<Log4NetMessage>>
{
  public override object Convert(IEnumerable<Log4NetMessage> obj)
  {
    var root = new TabSection("Level", "ThreadName", "LoggerName", "Message", "From Request Start", "From Last");
    foreach (var item in obj)
    {
      root.AddRow().
        Column(item.Level).
        Column(item.ThreadName).
        Column(item.LoggerName).
        Column(item.Message).
        Column(item.FromFirst).
        Column(item.FromLast).
        Style(GetStyle(item.Level));
    }
      return root.Build();
  }

  private static string GetStyle(string levelDisplayName)
  {
    switch (levelDisplayName)
    {
      case "EMERGENCY":
      case "FATAL":
      case "ALERT":
        return FormattingKeywords.Fail;

      case "CRITICAL":
      case "SEVERE":
      case "ERROR":
        return FormattingKeywords.Error;

      case "WARN":
        return FormattingKeywords.Warn;

      case "NOTICE":
      case "INFO":
        return FormattingKeywords.Info;

      case "DEBUG":
      case "FINE":
      case "TRACE":
      case "FINER":
      case "VERBOSE":
      case "FINEST":
      case "ALL":
        return FormattingKeywords.Quiet;

      default:
        return FormattingKeywords.Quiet;
    }
  }
}

Result

Finally you have to add the GlimpseAppender to your Log4Net configuration. When you start now your application, you should have a new Log4Net tab in the Glimpse window. Its content is similar to the Trace tab above, but the data is better structured.

NOTE: Be sure to start the application really from scratch. E.g. stop an already running IISExpress before. Otherwise it could be the Glimpse setup is not working as expected.

---

You can find the source code at GitHub: https://github.com/Ritzlgrmft/GlimpseDemo.

Web diagnostics with Glimpse

Glimpse is a great tool to get more info what is going on your web server. And it is easy to use: for my demo, I created a new ASP.NET MVC 4 Web Application (it works also with other MVC versions, and also with ASP.NET Web Forms) with the Internet Application template (again, it works also with the other templates). Then I added the NuGet package Glimpse.Mvc4. Finally I pressed F5, the application started and - nothing changed! No hint of Glimpse.
The reason is that Glimpse has to be enabled first. For this I opened the page Glimpse.axd in the root of my web application:

I clicked on Turn Glimpse On, went back to my application home page, refreshed it, and now in the bottom right corner was the Glimpse icon: . Clicking on this icon, I got the Glimpse window, displaying several tabs with some info used to produce the current web page:

NOTE: Glimpse stores the info if it is enabled or not in a cookie called glimpsePolicy. That means, when you later access the page again, Glimpse will be enabled already (or not).

When you look at the Glimpse tabs, some like Request, Server or Trace appear familiar from ASP.NET Tracing. But there are also some other tabs. Just play around with them a little bit.

Entity Framework Plugin

As useful as Glimpse is so far - that is not all. You can easily add additional tabs to Glimpse. A complete list of all available packages can be found on the Glimpse Extensions page.
The Internet Application template I used above contains some database access in combination with the user stuff (Log in, Register). Therefore I added now the Entity Framework Plugin.
I just had to download the NuGet package Glimpse.EF5. After compiling I refreshed the home page and found a new SQL tab in the Glimpse window. Only the tab was disabled since no database access was done for the home page.
Then I logged in, but again the SQL tab was disabled. This happened because there several requests executed for the log in, and only the last request didn't access the database. For problems like this Glimpse provides the History tab:

Here I selected the correct request, clicked Inspect, and finally the SQL tab was enabled:

Great, isn't it? A lot of useful info, and no effort to get them.

Self-host ASP.NET Web API and SignalR together

A few days ago, I wanted to build a Windows service providing some services via ASP.NET Web API. This was done easily. I just used the HttpSelfHostServer, and everything was fine.

But then I wanted also to inform the clients about some internal changes in my program in an asynchronous way. For this I thought ASP.NET SignalR would be the perfect solution. But unfortunately, SignalR expects an OWIN host, not the HttpSelfHostServer.

Then I spent some time with googling. And finally I was able to find the solution. I had to add the following NuGet packages to my project:

Microsoft.AspNet.WebApi.Owin
Microsoft.Owin.Host.HttpListener
Microsoft.Owin.Hosting

All packages are in pre-release state (version 0.21.0-pre at the moment). But they work already. To fire up the OWIN host, I needed only the following few lines of code:

const string serverUrl = "http://localhost:8080";
using (WebApplication.Start<Startup>(serverUrl))
{
 Thread.Sleep(Timeout.Infinite);
}

The configuration itself is done in the Startup class:

public class Startup
{
 public void Configuration(IAppBuilder app)
 {
  // Configure WebApi
  var config = new HttpConfiguration();
  config.Routes.MapHttpRoute(
   "API Default", "api/{controller}/{id}", new { id = RouteParameter.Optional });
  app.UseWebApi(config);

  // Configure SignalR
  app.MapHubs();
 }
}

Quite easy if you know it...