Patrick Matula
Posts About Me Notes Community Twitter LinkedIn
Patrick Matula

What is Citrix Diagnostic Facility (CDF)?

 included in  category Citrix
     1363 words   7 minutes 

If you work as Citrix Sysadmin and open a Citrix case you have a good chance that the Citrix support engineer tells you: “Please run CDFControl, check all modules and reproduce the issue while recording a trace”.

But what is it?

We approach the topic first by looking at the tools from Citrix.

CDFAnalyzer is a deprecated tool used to analyze ETL files. If you search, you’ll still find the binaries but there is no real value to having them. CDFAnalyzer expects that the TMF files are already available locally.

CDFControl is the tool to record and parse a CDF trace. If you have an ongoing case with Citrix you probably do only the record part. The UI is relatively simple. You need to start CDFControl as administrator, check the needed modules (or all modules in the dropdown menu), and click on “Start Tracing”. After reproducing the issue, you stop the tracing and CDFControl will create a folder with everything needed inside. In a Citrix support case, you will zip it and upload it to the support portal.
But it’s also possible to analyze the recorded CDF trace. How? File -> Parse Trace and select the recorded CDF trace. The good thing (compared with CDFAnalyzer) is that CDFControl will download only the needed TMF files automatically.

CDFMonitor is a tool to record a CDF Trace. You may wonder why CDFMonitor is needed when CDFControl can record and parse a CDF trace. The reason is that CDFControl is a GUI tool but sometimes you want to collect CDF traces in a script / programmatically. You need two things: CDFMonitor.exe and CDFMonitor.exe.config. The config file should have configured a few things, an (partial) example would be:

<add key="debug" value= "True" />
<add key="logfileautoflush" value="True" />
<add key="logfilemaxcount" value="20" />
<add key="logfilemaxsize" value="100" />
<add key="logfilename" value="X:\cdfmonitor.csv" />
<add key="logfileoverwrite" value="True" />
<add key="logtoetl" value="True" />
<add key="tmfserver" value= "http://ctxsym.citrix.com/tmfs/xaxd/" />
<add key="tracefile" value="X:\cdfmonitor.etl" />

There is a high chance that the Citrix support engineer needs specific values.
When you have these two files, you may do the following things to setup CDFMonitor.exe:

CDFMonitor.exe /installservice
CDFMonitor.exe /startservice
{reproduce the issue / wait until the issue happens}
CDFMonitor.exe /stopservice

Wait until the service stopped properly and after that, you can grab the file, zip it and upload it to the Citrix support portal.
CDFMonitor is useful in scenarios when you have hundreds of VDAs, the issue is completely random and there is no chance to reproduce the issue. With CDFMonitor you can install the CDFMonitor (if you use PVS, just install it in the master image), roll out the image and when the user calls you, you connect to the affected VDA and do the /stopservice and grab the files.

I wrote a few times about “tmf files” but what is it? Especially if you try to analyze the CDF traces by yourself, you can see something like this: readable CDF trace Everything is good, you can read all the lines and try to figure out what happens. But sometimes you’ll also see this: unreadable CDF trace You see only “TMF file not found”. So what, we can say is that TMF files are needed to convert binary messages to readable messages. You’ll find out more about TMF files here: https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/trace-message-format-file
How do you get the TMF files? The CDFControl will download them automatically from http://ctxsym.citrix.com/symbols.
This means that you are heavily dependent on Citrix to provide the TMF files. Some product groups still provide TMF files (like delivery controller team does a pretty good job), but unfortunately, some no longer do.

Warning
The section about TMF files seems to be wrong. It is even readable without any internet connection. You can read the Twitter conversation with Guy Leech below. tldr: It’s very likely that there are no (or useless) .pdb/.tmf files available on the Citrix symbol server. How does reading the messages work then? I have no idea. If someone can enlighten me about it, gladly.

.. and what is CDF now?

Bas van Kaam is the author of the book “Inside Citrix: The Flexcast Management Architecture” and the book gives a very good overview of various topics. After a time, Bas van Kaam released the whole book online (which is awesome) and you can read about CDF traces here: https://www.basvankaam.com/inside-citrix-chapter-twenty-three-the-one-with-all-the-troubleshooting/.
CDF is built on Event Tracing for Windows (ETW). You’ll find information about ETW here: https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/event-tracing-for-windows--etw- and here: https://docs.microsoft.com/en-us/windows/win32/etw/about-event-tracing.
If you want to dive a little bit deeper you should read the blogpost by Nasreddine Bencherchali: A Primer On Event Tracing For Windows (ETW).

Okay, let’s do the obvious thing and get the ETW providers from Citrix. Delivery Controller:

> logman query providers
...
Citrix AD Identity Service               {676A0E9D-FEA1-547C-C303-2379E2C49818}
Citrix Analytics                         {BD423FCF-3008-7660-7BA0-94CE15772963}
Citrix App Library                       {30094DEC-E769-9593-5313-EB936B233F76}
Citrix Broker Service                    {1EC1549E-1762-49AB-B7A8-0DE5CBACA3FB}
Citrix ConfigSync Service                {A137169A-EEE0-4AE7-A5A3-11905BEED74D}
Citrix Configuration Logging Service     {3C7AF5B4-C4BF-0AF6-B6A0-D9E53BEF8A36}
Citrix Configuration Service             {18F6C974-03EB-3283-B33A-DB8BB62761FA}
Citrix Delegated Administration Service  {EF06B477-25A4-92B8-6FDA-901D692B27B9}
Citrix Environment Test Service          {0A34596B-7C0F-E9D5-20D8-71E329C7B661}
Citrix High Availability Service         {55CCD184-A006-4EC2-9B0A-E6BDAA4D19B7}
Citrix Host Service                      {32760596-85DD-C318-8EF1-74BECB570AED}
Citrix Machine Creation Service          {61302E91-3CC9-A923-866A-D734E2A68CE3}
Citrix Monitor Service                   {BC6740BD-CB9E-FD5A-C5DF-3A0D024C058C}
Citrix Orchestration Service             {F946495F-0D3B-72FF-C787-38B1CC9EC9D2}
Citrix Storefront Service                {9363EC39-1D56-F505-98D4-74FDBAC4EB0D}
Citrix Trust Service                     {CE889E69-6BD6-17F8-8A2F-1441EBE06291}
Citrix-Broker                            {D062513E-0D1F-4033-8CA0-B2AC667B3DB8}
Citrix-XenDesktop-BrokerMonitor          {2386F16F-1B93-4F2B-91A8-0DE5ADFA092A}
Citrix-XenDesktop-MCSMonitor             {CD220B38-A589-41EA-83FF-B759829E4B6E}
...

PVS Server:

> logman query providers
...
Citrix-CDF-ErrorReporter                 {E074C2C9-FB9D-49D1-BD50-F76F01AC1D92}
Citrix-VHD-PVS                           {08E5FBC9-3ADA-488C-BB19-B9A747E3C938}
...

VDA:

> logman query providers
... 
Citrix-AppExperience-Seamless            {2025A819-4EA0-47C1-87DB-2C62CA9DF425}
Citrix-CDF-ErrorReporter                 {E074C2C9-FB9D-49D1-BD50-F76F01AC1D92}
Citrix-Device-Redirector                 {91C52BC6-1F18-4D5F-9A02-67957DF20096}
Citrix-Multimedia-AudioSvc               {A550AA40-B443-468F-9FC4-29D27E0F6840}
Citrix-Multimedia-BCR                    {2E67EA23-CFAE-4AE9-8F8C-DF95DD38A695}
Citrix-Multimedia-Rave                   {814A62B9-61E9-41CC-BBDB-8086A3F9804C}
Citrix-VHD-PVS                           {08E5FBC9-3ADA-488C-BB19-B9A747E3C938}
...

Storefront: none.
License server: none.
Citrix WEM: none.

I guess a few sound familiar to you because some of them are also displayed in the EventLog.

You see the delivery controller provides 19 ETW providers but when you open the CDFControl you’ll see way more. Why?
Thereby applies: “When in doubt, run Process Monitor!”. And that’s exactly what I did. After a few seconds you’ll find the following registry path:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Citrix\Tracing\Modules
It contains several subkeys with a name (the name for the display ETW provider). Then you have several values, something like that: BrokerDAL registry

For now, the interesting part is the GUID.. and if we check the GUID with the list above, we’ll not find it. Why?
The reason is that logman query providers only list providers when they’re registered https://docs.microsoft.com/en-us/windows/win32/wes/developing-a-provider and https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/tracelog-enumguid-display But it’s not a duty to do so.

To sum up, Citrix provides way more ETW providers, and CDFControl recognizes them, because of the registry key above.

Is it possible that Citrix has additional ETW providers but they don’t disclose them? Yes. How to find out? logman query providers doesn’t help.. so one way is to check every running process from Citrix (the command would be: logman query providers -pid 9999). You get a list of different unnamed ETW providers, then you need to compare them with the list we get from HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Citrix\Tracing\Modules and all that remains are then unknown ETW providers. Unknown in this context means neither CDFControl nor Citrix tells us what this could be nor do we have any reference to it.

To get these unknown ETW providers I tried to catch up with a few processes and didn’t find any specific ETW provider for a specific Citrix-related process. But I only checked ten processes or so… There is also another thing that I can’t be sure that an unnamed ETW provider is from Citrix (could be also from another component/vendor). So no 100% solution, I guess.


If you have some time, you can subscribe to the unknown ETW providers and look at the data (hopefully there are PDB or TMF to translate the binary data…).


There is also a possibility in CDFControl to import a .CTL file. The CTL file (and you can open it with a simple text editor) provides exactly that information: GUID and a useful name of a ETW provider. An example would be: Special CTL for PVS diagnostic tracing.

I hope you had and will have some fun with CDF tracing / analyzing and poking around with ETW. I had some fun making a quick write-up about the topic. And as always: Please let me know if you have any comments, tips, or similar for me.

Updated on 2022-07-06
Back | Home