Certain System.* not being embedded?

Question

Certain System.* not being embedded?

EricZimmerman opened this issue 3 years ago · comments

Hello!

I am an active supporter of Fody. i am using costura v5.0.2

I noticed recently that certain things like the following do not seem to be getting rolled into the exe like they used to.

System.Text.Encoding.CodePages.dll

this leads end users to run into issues when they use the software. I do see plenty of other System.* stuff in the obj folder tho

if I tweak my xml config with this

<Costura>
  <IncludeAssemblies>
    System.Text.*
    AlphaFS
CsvHelper
Exceptionless
FluentCommandLineParser
GuidMapping
Microsoft.Win32.Primitives
NFluent
NLog
RawCopy
Registry
ServiceStack.Text
ShellBagsParser
System.Buffers
System.Numerics.Vectors
System.Runtime.CompilerServices.Unsafe
  </IncludeAssemblies>
    </Costura>

I CAN see this file here and compressed, as expected:

obj\Release\Costura

but if I have a default xml config, that file does not show up in obj\Release\Costura

I also always seem to get this warning:

Fody: The package reference for Costura.Fody is missing the 'compile' part in the IncludeAssets setting; it's recommended to completely remove IncludeAssets

which i remove but it does not seem to help.

I saw #652 and thought that would do it. i assume there is not a newer version out there later than 5.0.2?

any idea why this is happening and how to fix it?

Geert van Horrik · Answer 1 · Wed Mar 17 2021 17:50:35 GMT+0800 (China Standard Time)

5.0.2 is the latest version available.

What version of .NET are you targeting?
How are you using Fody & Costura in your csproj file?

Eric · Answer 2 · Wed Mar 17 2021 20:01:25 GMT+0800 (China Standard Time)

.net 4.6.2 to maybe 4.7 here and there, but the projects i have seen this recently are 4,6 based. Not doing anything .net core with fody\custura
its as simple as it gets, and whatever happens when nuget adds them to the project. one looks like this (other stuff removed). I did remove the IncludeAssets tag per the warning. I always had an empty FodyWeavers.xml file as well, but only recently started experimenting with having to add the names via IncludeAssemblies. usually its just

<ItemGroup>
    <PackageReference Include="Costura.Fody">
      <Version>5.0.2</Version>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>

    <PackageReference Include="Fody">
      <Version>6.4.0</Version>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
  
  </ItemGroup>

Eric · Answer 3 · Wed Mar 17 2021 20:15:00 GMT+0800 (China Standard Time)

btw i am using Version 16.9.1 of VS Pro

here is another example from a different project, showing the defaults after adding from nuget:

 <ItemGroup>
    <PackageReference Include="Costura.Fody">
      <Version>5.0.2</Version>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="CsvHelper">
      <Version>26.0.1</Version>
    </PackageReference>
    <PackageReference Include="Exceptionless">
      <Version>4.6.2</Version>
    </PackageReference>
    <PackageReference Include="FluentCommandLineParser">
      <Version>1.4.3</Version>
    </PackageReference>
    <PackageReference Include="Fody">
      <Version>6.4.0</Version>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="MaxMind.GeoIP2">
      <Version>4.0.1</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.CSharp">
      <Version>4.7.0</Version>
    </PackageReference>
    <PackageReference Include="NLog">
      <Version>4.7.8</Version>
    </PackageReference>
  </ItemGroup>

this too results in files remaining in my debug folder on compile:

System.IO.Compression.dll
System.Net.Http.dll
System.Runtime.InteropServices.RuntimeInformation.dll
System.Security.Cryptography.Algorithms.dll
System.Security.Cryptography.Encoding.dll
System.Security.Cryptography.X509Certificates.dll

Geert van Horrik · Answer 4 · Wed Mar 17 2021 21:10:33 GMT+0800 (China Standard Time)

If you view your assembly with IL Spy, you should see the costura embedded info under resources. Can you please post that as well?

Eric · Answer 5 · Wed Mar 17 2021 21:26:22 GMT+0800 (China Standard Time)

this is for the last example i did.

in my quick review, i do not see reference to System.Security.Cryptography.X509Certificates.dll at least

Geert van Horrik · Answer 6 · Wed Mar 17 2021 21:51:15 GMT+0800 (China Standard Time)

Can you check costura.metadata contents and post it here?

Geert van Horrik · Answer 7 · Wed Mar 17 2021 21:52:35 GMT+0800 (China Standard Time)

And it would be best if there is some sort of repro so I can try to reproduce. How is System.Security.Cryptography.X509Certificates getting there without being referenced (so probably there are some implicit dependencies that Costura is not picking up correctly).

Geert van Horrik · Answer 8 · Wed Mar 17 2021 21:53:02 GMT+0800 (China Standard Time)

And, one more thing: you can enable debug logging (verbose) for ms build and see what Costura logs to the output window, it might give some useful hints.

Eric · Answer 9 · Wed Mar 17 2021 22:53:06 GMT+0800 (China Standard Time)

use this repo. same one i am working with above. this is not the same project i mentioned initially about codepages, but i am guessing its all related. i suppose adding a ref to

System.Text.Encoding.CodePage

would let you see that happen in this project as well.

https://github.com/EricZimmerman/iisGeolocate

as to where the 509 thing is coming from, not sure, but i know i have nuget with the codepages reference in it and that dll gets dropped too.

costura.metadata

costura.costura.dll.compressed|5.0.2.0|Costura, Version=5.0.2.0, Culture=neutral, PublicKeyToken=null|Costura.dll|DC4B6E415DF4BAFDA5C2A6228D465D64383A8DAF
costura.costura.pdb.compressed|||Costura.pdb|51DBB96092007F8A2F4551B197DA996E4B48C217
costura.csvhelper.dll.compressed|26.0.0.0|CsvHelper, Version=26.0.0.0, Culture=neutral, PublicKeyToken=8c4959082be5c823|CsvHelper.dll|87A29163204E9FEFF52B6D6E6815ACD749F35A62
costura.exceptionless.dll.compressed|4.0.0.0|Exceptionless, Version=4.0.0.0, Culture=neutral, PublicKeyToken=fc181f0a46f65747|Exceptionless.dll|FC800C27F4D5F9DD0386389327B7F4FF206EEA09
costura.exceptionless.pdb.compressed|||Exceptionless.pdb|617CBFA2CC444A95ED3DF98FCE5A36E4BFC9054F
costura.fluentcommandlineparser.dll.compressed|1.4.3.0|FluentCommandLineParser, Version=1.4.3.0, Culture=neutral, PublicKeyToken=null|FluentCommandLineParser.dll|553EC4ECC90676C7BB1DE9F75A6B1226F39677AA
costura.fluentcommandlineparser.pdb.compressed|||FluentCommandLineParser.pdb|5753997682C6B57B8F8A946382220492D45665BD
costura.maxmind.db.dll.compressed|2.0.0.0|MaxMind.Db, Version=2.0.0.0, Culture=neutral, PublicKeyToken=66afa4cc5ae853ac|MaxMind.Db.dll|96DB62E4BC50F390EAD70AAAF8A9942B2EA857DF
costura.maxmind.geoip2.dll.compressed|4.0.0.0|MaxMind.GeoIP2, Version=4.0.0.0, Culture=neutral, PublicKeyToken=66afa4cc5ae853ac|MaxMind.GeoIP2.dll|46CC26D014261DA66DE45CECF9BEAB44ABCE0566
costura.microsoft.bcl.asyncinterfaces.dll.compressed|5.0.0.0|Microsoft.Bcl.AsyncInterfaces, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51|Microsoft.Bcl.AsyncInterfaces.dll|5DBF1534FFBFF016CC45559EB5EFF3DC4252A522
costura.microsoft.win32.primitives.dll.compressed|4.0.2.0|Microsoft.Win32.Primitives, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|Microsoft.Win32.Primitives.dll|AFA308C0166D8CBDD1D5E6FDAA8C9B87EB9184E7
costura.nlog.dll.compressed|4.0.0.0|NLog, Version=4.0.0.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c|NLog.dll|13E601EC1253B135BC4E5B64442AB443DB172E0E
costura.system.appcontext.dll.compressed|4.0.0.0|System.AppContext, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.AppContext.dll|C9A0C7200AD55960A1E1824B04718CBF6CA84581
costura.system.buffers.dll.compressed|4.0.3.0|System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51|System.Buffers.dll|2F410A0396BC148ED533AD49B6415FB58DD4D641
costura.system.collections.immutable.dll.compressed|1.2.5.0|System.Collections.Immutable, Version=1.2.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Collections.Immutable.dll|88DF86EE0B137992AF15A35825804274FA252E30
costura.system.console.dll.compressed|4.0.1.0|System.Console, Version=4.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Console.dll|B5102629939EBDD4BCD0F407A92828DEE5F2B565
costura.system.diagnostics.diagnosticsource.dll.compressed|4.0.1.0|System.Diagnostics.DiagnosticSource, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51|System.Diagnostics.DiagnosticSource.dll|85DC92EDD4B0049ED9049E075C4DEF8A3D64E43B
costura.system.diagnostics.tracing.dll.compressed|4.1.1.0|System.Diagnostics.Tracing, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Diagnostics.Tracing.dll|5340B1FC77E793D1FDDE49B14B7B426D0E448870
costura.system.globalization.calendars.dll.compressed|4.0.2.0|System.Globalization.Calendars, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Globalization.Calendars.dll|30AE8C8E57BCABE27BA675FA84D4DA6AF3C10EAD
costura.system.io.dll.compressed|4.1.1.0|System.IO, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.IO.dll|BA5F2CAA404AE9A283A841C08B024BAF8817BA01
costura.system.io.compression.zipfile.dll.compressed|4.0.2.0|System.IO.Compression.ZipFile, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b77a5c561934e089|System.IO.Compression.ZipFile.dll|99D6C187DE8510206A93D2EED9C65E65E0C86E72
costura.system.io.filesystem.dll.compressed|4.0.2.0|System.IO.FileSystem, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.IO.FileSystem.dll|B46AD93384323ED4AA88E8AF3CDEE208A88F1F6D
costura.system.io.filesystem.primitives.dll.compressed|4.0.2.0|System.IO.FileSystem.Primitives, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.IO.FileSystem.Primitives.dll|8374975CA07A300021EF0ACAA4F908D5BFB102FB
costura.system.memory.dll.compressed|4.0.1.1|System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51|System.Memory.dll|19BF79600B716523B5317B9A7B68760AE5D55741
costura.system.net.sockets.dll.compressed|4.1.1.0|System.Net.Sockets, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Net.Sockets.dll|00278D53E59026CD8C22889773466FEF27D52BBF
costura.system.numerics.vectors.dll.compressed|4.1.4.0|System.Numerics.Vectors, Version=4.1.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Numerics.Vectors.dll|3D216458740AD5CB05BC5F7C3491CDE44A1E5DF0
costura.system.reflection.dll.compressed|4.1.1.0|System.Reflection, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Reflection.dll|B830E365EFC12167714E591DC7C1DD0A623FCA48
costura.system.reflection.metadata.dll.compressed|1.4.5.0|System.Reflection.Metadata, Version=1.4.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Reflection.Metadata.dll|6F0F1BA6DF9379DDC9DE1D586286F621AF69C298
costura.system.runtime.dll.compressed|4.1.1.0|System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Runtime.dll|7E549944512DE844B49AAD4B9D4A9547C06EDE37
costura.system.runtime.compilerservices.unsafe.dll.compressed|5.0.0.0|System.Runtime.CompilerServices.Unsafe, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Runtime.CompilerServices.Unsafe.dll|BAB44612721C3DC91AC3D9DFCA7C961A3A511508
costura.system.runtime.extensions.dll.compressed|4.1.1.0|System.Runtime.Extensions, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Runtime.Extensions.dll|9C6772A60E6464208A491A2CB55D62C8F3C67D00
costura.system.runtime.interopservices.dll.compressed|4.1.0.0|System.Runtime.InteropServices, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Runtime.InteropServices.dll|452039516E53E284D9DB5112CD1D5D50A954C1E2
costura.system.security.cryptography.primitives.dll.compressed|4.0.1.0|System.Security.Cryptography.Primitives, Version=4.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Security.Cryptography.Primitives.dll|62A622557CC0D6FCED9C1A14BE28DBC39E9BD6FC
costura.system.text.encodings.web.dll.compressed|5.0.0.0|System.Text.Encodings.Web, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51|System.Text.Encodings.Web.dll|23A0B63A62A6F0907DAF759088A2A489990FCF2F
costura.system.text.json.dll.compressed|5.0.0.0|System.Text.Json, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51|System.Text.Json.dll|DF505C2FF55722D3F552D4A7A72C9FCADFB5B60C
costura.system.threading.tasks.extensions.dll.compressed|4.2.0.1|System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51|System.Threading.Tasks.Extensions.dll|2242627282F9E07E37B274EA36FAC2D3CD9C9110
costura.system.valuetuple.dll.compressed|4.0.3.0|System.ValueTuple, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51|System.ValueTuple.dll|81CD1ACB53D493C54C8D56F379D790A901A355AC
costura.system.xml.readerwriter.dll.compressed|4.1.0.0|System.Xml.ReaderWriter, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a|System.Xml.ReaderWriter.dll|765C9E4D36B4F2986514C1AB24B5E5E56AFFF448

diagnostic build

buildinfo.txt

Cédric Luthi · Answer 10 · Fri Mar 19 2021 18:05:19 GMT+0800 (China Standard Time)

Costura 5 has changed the way it embeds packages which have runtime assemblies. It is still possible to handle those System.* assemblies with Costura 5 but it involves a lot more work than with Costura 4. I have written a full explanation with sample code in #651 (comment). The pain point is that you have to write your own AssemblyResolve callback to make it work now. 🙁

This is very unfortunate, especially since the dll files which are inside the runtimes directories are byte for byte equivalent to the libraries inside the lib directories. (At least for all those that I have tested.)

comp %USERPROFILE%\.nuget\packages\system.text.encoding.codepages\5.0.0\runtimes\win\lib\net461\System.Text.Encoding.CodePages.dll %USERPROFILE%.nuget\packages\system.text.encoding.codepages\5.0.0\lib\net461\System.Text.Encoding.CodePages.dll
Files compare OK

comp %USERPROFILE%\.nuget\packages\system.security.cryptography.x509certificates\4.3.0\runtimes\win\lib\net461\System.Security.Cryptography.X509Certificates.dll %USERPROFILE%.nuget\packages\system.security.cryptography.x509certificates\4.3.0\lib\net461\System.Security.Cryptography.X509Certificates.dll
Files compare OK

I'm using Costura on .NET Framework exclusively (I use single file executables on .NET Core) and I'd prefer to have Costura 4 behavior when it comes to embedding runtime references.

@GeertvanHorrik, would you accept a pull request that restores the Costura 4 behavior on .NET Framework assemblies?

Geert van Horrik · Answer 11 · Fri Mar 19 2021 18:15:53 GMT+0800 (China Standard Time)

For full .NET => yes, for .NET Core => no

Eric · Answer 12 · Fri Mar 19 2021 20:31:52 GMT+0800 (China Standard Time)

this would be most welcome. i too have a lot of "legacy" .net stuff and would benefit from this.

Cédric Luthi · Answer 13 · Fri Mar 19 2021 20:34:37 GMT+0800 (China Standard Time)

I'm working on it! Gotta finish my GitHub sponsoring page. 😉

Cédric Luthi · Answer 14 · Fri Mar 19 2021 22:13:05 GMT+0800 (China Standard Time)

The pull request is ready for review: #679. It basically restores the Costura 4 behavior for embedding runtime assemblies on .NET Framework while preserving the new behavior for .NET Core. All this is configurable through the new UseRuntimeReferencePaths configuration setting.

Geert van Horrik · Answer 15 · Sat Mar 20 2021 20:16:59 GMT+0800 (China Standard Time)

@0xced did a really good job here, well done! Will release a prerelease now. Please let us know if it's all good so we can release 5.1.

Eric · Answer 16 · Sat Mar 20 2021 21:05:24 GMT+0800 (China Standard Time)

ill give it a try. based on the change, it sounds like plain old

<Costura/>

will do the trick, like in days of old

Geert van Horrik · Answer 17 · Sat Mar 20 2021 21:24:09 GMT+0800 (China Standard Time)

Correct. The version is here: https://www.nuget.org/packages/Costura.Fody/5.1.0-alpha0409

Eric · Answer 18 · Sat Mar 20 2021 21:28:07 GMT+0800 (China Standard Time)

nature is healing!

it works! i can see the difference in the size of the final exe too and binary runs as expected.

the warning about *Assets is now gone too

Cédric Luthi · Answer 19 · Sat Mar 20 2021 22:01:30 GMT+0800 (China Standard Time)

I can see that there's still a pdb and a config file containing only supported runtime and binding redirects. If you want to get rid of the pdb as a separate file but still want the stack trace with source code lines information you can use the following setting in your csproj file. I like to do this to ensure that the pdb matches the exe so that I'm 100% sure the line numbers in stack traces are correct.

<PropertyGroup>
  <DebugType>embedded</DebugType>
</PropertyGroup>

And if you want to get rid of the config file, delete it and add the following settings in your csproj file so that it's not automatically recreated:

<PropertyGroup>
  <AutoGenerateBindingRedirects>false</AutoGenerateBindingRedirects>
  <GenerateSupportedRuntime>false</GenerateSupportedRuntime>
</PropertyGroup>

Binding redirects are useless when the dll is embedded with Costura and I think it's not a high risk to remove the supported runtime setting which target .NET Framework 4.6.2. I'm not really sure what this does, I think it might prompt the user to install the required version of .NET framework if it's not installed.

Eric · Answer 20 · Sat Mar 20 2021 22:14:14 GMT+0800 (China Standard Time)

that will be awesome. would love to have actual line numbers in release.

Eric · Answer 21 · Sat Mar 20 2021 22:20:34 GMT+0800 (China Standard Time)

after reloading the project, the warning is back

Fody: The package reference for Costura.Fody is missing the 'compile' part in the IncludeAssets setting; it's recommended to completely remove IncludeAssets

any reason why this cant be fixed somehow? what exactly does this even mean? does not seem to harm anything if its there, or not.

Cédric Luthi · Answer 22 · Sat Mar 20 2021 22:26:11 GMT+0800 (China Standard Time)

that will be awesome. would love to have actual line numbers in release.

You will, even if you have to use CreateTemporaryAssemblies='true'. 😇 Have a look at #633 if you like gory technical details. 😉

the warning is back

Here’s how to fix it (remove the IncludeAssets element):

<ItemGroup>
  <PackageReference Include="Costura.Fody">
    <Version>5.1.0-alpha0409</Version>
    <PrivateAssets>all</PrivateAssets>
  </PackageReference>
</ItemGroup>

Eric · Answer 23 · Sat Mar 20 2021 22:27:50 GMT+0800 (China Standard Time)

yep, thats what i have been doing, removing it.

Geert van Horrik · Answer 24 · Sat Mar 20 2021 23:11:38 GMT+0800 (China Standard Time)

Just published 5.1 stable.

Geert van Horrik · Answer 25 · Sat Mar 20 2021 23:11:50 GMT+0800 (China Standard Time)

https://www.nuget.org/packages/Costura.Fody/5.1.0

Cédric Luthi · Answer 26 · Sat Mar 20 2021 23:20:16 GMT+0800 (China Standard Time)

that will be awesome. would love to have actual line numbers in release.

Actually, you need .NET Framework 4.7.2 to get line numbers with portable/embedded pdb, see Stack traces are missing source information for frames with debug information in the Portable PDB format when running on .NET Framework 4.7.1 and Retargeting Changes for Migration from .NET Framework 4.7.1 to 4.7.2

Cédric Luthi · Answer 27 · Sat Mar 20 2021 23:23:48 GMT+0800 (China Standard Time)

Thanks for releasing 5.1.0 stable, Geert. Could you please also create releases on Costura's GitHub releases page, ideally writing some release notes in them?

Geert van Horrik · Answer 28 · Wed Apr 07 2021 19:08:35 GMT+0800 (China Standard Time)

The deployment pipeline should automatically create the releases (drafts) with the tickets if we were created the milestones and added the tickets there. Now it's a manual process, will see what we can do about htis in the future.

Geert van Horrik · Answer 29 · Wed Apr 07 2021 19:19:27 GMT+0800 (China Standard Time)

Created milestones for 5.2.0 and assigned the tickets. It should auto create the release notes next time.