Some quick notes on Microsoft Copilot
This is being archived here. I initially posted this on Twitter.
I've had a surprising amount of people ask me about Copilot and the stick I'm poking it with. Copilot is a hot topic, so I assume people are genuinely interested in how it works?
I can't really give a good tl;dr because I'm still poking it with a stick. There is a lot of stuff I don't quite understand (as is tradition), so I can only share some of my insights and speculations.
Copilot.exe (the main binary) is just a .NET runtime host. MSDN has some articles about it. Basically the .exe you execute does a bunch of fancy shit, it modifies some stuff in the .exe itself (Thread Environment Block) for custom error handling to be all fancy, or whatever. It eventually invokes the Windows Library Core Language Runtime library (libcoreclr) function "coreclr_execute_assembly" and the "real" Copilot runs from Copilot.dll.
Copilot.dll (I'll just call it Copilot, whatever) is a big ass fuck off C#.NET application with what feels like over 9000 dependencies and libraries. It's a big heavy bloated son of a bitch.
Copilot determines the .NET version it's supposed to run on from a JSON file in the current directory titled "runtimeconfig.json".
Copilot uses Microsoft UI Xaml (WinUI 3?) so it is ridiculously heavy and feels like it lags constantly.
Copilot does all AI stuff server side at Microsoft at "copilot-dot-microsoft-dot-com/c/api". It looks* like it authenticates to the Copilot servers using the Microsoft account you make when you first setup Windows 11. It looks like it may also support Apple and Google, but I haven't poked it enough.
Every action taken in Copilot is a "view" and goes through a URI thingy. It's some C#.NET bullshit. I barely understand it. You can easily see all the different "views" and the URI it goes through in Copilot to load different "views" (different parts of Copilot?)
Even simple acts as viewing a different "view", scrolling up to see previous messages sent to Copilot, etc. all go through API requests to Microsoft. It is all stored over on their stuff. Hence, Copilot can feel ridiculously slow when scrolling up to review message history. It goes through stuff like "GetConversationHistoryEndpoint" inside of CopilotNative.Platform (1.25111,85.0 .NETCoreApp, v9.0).
So... anything you do is going to through their web API. It slows things down dramatically. Even renaming a conversation makes a web call.
Also, anytime you send a message to Copilot it goes through a fucking MASSIVE nested procedure that bounces all through all the dependencies. However, this is pretty standard stuff for big .NET applications.
To make a long story short-ish, each message you to Copilot is tokenized (or rather, placed into a "Dictionary"). This dictionary contains the data you're sending and any files you're attaching. Part of this process Copilot makes a very minor attempt at sanitizing data for "anonymity".
Copilot has different stuff in place for removing data and sensitive information but the actual act of sending a message to Copilot only censors file paths from your machine (if you send a file). In other words,
C:\Users\TommyPoop\File.txt
transforms into ..
C:\Users\<redacted>\File.txtI haven't seen anywhere else where this logic is implemented, but it probably does more stuff somewhere. I doubt they'd include all this PII censoring logic for no reason.
Copilot also has stuff in place for advertisement identifiers, health and fitness, shopping habits, etc. I'm not sure what that's all about. I also see the gaming stuff but I haven't poked that yet either.
Copilot also also has a bunch of stuff for PicassoAI for "PicassoLabs", "PicassoFinance", "PicassoBriefings". I don't know if this is a 3rd party thing or something they made internally. I have no idea what I'm looking at.
Anyway, that is my scattered thoughts on Copilot. It is basically a really, really, really fancy web browser that can only be used to communicate with Microsoft's AI endpoints.
I quickly realized though that if you go to
... and make an entry that makes the Microsoft Copilot AI domain resolve to localhost, Copilot implodes and drops dead. It can no longer access any API endpoints hence it cannot exist.
Last updated