New information. Pursuing the clr.dll crash error has led me on an interesting path. It looks like there is a bug in .NET 4.0 clr.dll that can cause this and applying KB2640103 has helped, but not 100%. Now my syncs only occasionally experience this issue.
I surveyed my other working servers and none have .net 4.0 installed.
Today it crashed when manually running delta sync on that MA and along with the original event log, a new .NET Runtime eventlog was created (no doubt due to additional logging provided by the update). Event ID was 1023
Application: miiserver.exe - Framework Version: v4.0.30319 - Description: The process was terminated due to an internal error in the .NET Runtime at IP 000007FEED00F670 (000007FEECD90000) with exit code 80131506.
Further research pointed to a connect.microsoft.com bug http://connect.microsoft.com/VisualStudio/feedback/details/696561/internal-error-in-the-net-runtime-exit-code-80131506
Microsoft stated: The issue you are seeing here is caused by managed heap corruption. This can be due to a wide variety of issues, such as an incorrect PInvoke signature, native code in your program corrupting the managed heap, or in rare cases a bug in CLR itself. For some background information, there are several places on the internet with information about debugging this type of issue:http://blogs.msdn.com/b/tess/archive/2006/02/09/net-crash-managed-heap-corruption-calling-unmanaged-code.aspx
This led me to http://support.microsoft.com/kb/2679415 which states:This bug can be encountered when the Garbage Collector is freeing and compacting memory. The error can happen when the Concurrent Garbage Collection is enabled and a certain combination of foreground Garbage Collection and background Garbage Collection occurs. When this situation happens you will see the same call stack over and over. On the heap you will see one free object and before it ends you will see another free object corrupting the heap.
I have therefore followed the instructions at http://msdn.microsoft.com/en-us/library/at1stbec.aspx adding <gcConcurrent enabled="false"/> to the miiserver.exe.config file under the configuration\runtime node. I restarted the service and we'll see if that finally resolves this issue.