Unloading of plctag.dll stuck in Shutdown

Question

Unloading of plctag.dll stuck in Shutdown

Teldaril opened this issue a year ago · comments

Hello,

i have a problem with the shutdown of the library. Probably it is more related to the c-library than to the c#-wrapper. But currently i am working with the wrapper. Here is a sample to reproduce the behavior:

var tag = new Tag<IntPlcMapper, short>()
{
    Name = "Test",
    Gateway = "0.0.0.0",
    PlcType = PlcType.Slc500,
    Protocol = Protocol.ab_eip,
    Timeout = TimeSpan.FromSeconds(1),
    DebugLevel = DebugLevel.Spew
};

try
{
    tag.Read();
}
catch (Exception e)
{
    Console.WriteLine(e);
}

Thread.Sleep(1000);

//Everything works fine when this is uncommented
//LibPlcTag.Shutdown();

The problem happens when i rely on the shutdown when the plctag.dll is unloaded after leaving the Program.cs. On Windows 7 i can get a stuck process blocking the plctag.dll from further usage. (e.g. restarting the given sample)

When running the sample on Windows 7 and Windows 10 with the preemptive Shutdown() uncommented the log looks like this:

2023-07-18 09:25:14.959 thread(2) tag(0) SPEW cond_wait_impl:1125 Starting. Called from tag_tickler_func:634.
...
2023-07-18 09:25:15.059 thread(7) tag(0) INFO plc_tag_shutdown:1095 Starting.
...
2023-07-18 09:25:15.059 thread(7) tag(0) INFO lib_teardown:176 Tearing down library.
...
2023-07-18 09:25:15.059 thread(7) tag(0) INFO lib_teardown:181 Signaling tag tickler condition var.
2023-07-18 09:25:15.059 thread(7) tag(0) SPEW cond_signal_impl:1192 Starting.  Called from lib_teardown:182.
2023-07-18 09:25:15.059 thread(7) tag(0) SPEW cond_signal_impl:1208 Done for call at lib_teardown:182.
2023-07-18 09:25:15.059 thread(7) tag(0) INFO lib_teardown:186 Tearing down tag tickler thread.
2023-07-18 09:25:15.059 thread(2) tag(0) SPEW cond_wait_impl:1148 Condition var wait returned.
2023-07-18 09:25:15.059 thread(2) tag(0) SPEW cond_wait_impl:1171 Condition var signaled for call at tag_tickler_func:634.
2023-07-18 09:25:15.059 thread(2) tag(0) SPEW cond_wait_impl:1182 Done for call at tag_tickler_func:634.
...
2023-07-18 09:25:15.059 thread(2) tag(0) INFO tag_tickler_func:646 Terminating.
2023-07-18 09:25:15.059 thread(7) tag(0) INFO lib_teardown:193 Tearing down tag tickler condition var.
...
2023-07-18 09:25:15.059 thread(7) tag(0) INFO lib_teardown:212 Done.

The tag_tickler_func thread is signaled by cond_signal_impl and received by cond_wait_impl. The tag_tickler_func thread correctly terminates -> 2023-07-18 09:25:15.059 thread(2) tag(0) INFO tag_tickler_func:646 Terminating.

When running the sample on Windows 10 with the preemtive Shutdown() commented it looks like this:

2023-07-18 08:51:36.007 thread(2) tag(0) SPEW cond_wait_impl:1125 Starting. Called from tag_tickler_func:634.
...
2023-07-18 08:51:36.091 thread(6) tag(0) INFO plc_tag_shutdown:1095 Starting.
...
2023-07-18 08:51:36.091 thread(6) tag(0) INFO lib_teardown:176 Tearing down library.
...
2023-07-18 08:51:36.091 thread(6) tag(0) INFO lib_teardown:181 Signaling tag tickler condition var.
2023-07-18 08:51:36.091 thread(6) tag(0) SPEW cond_signal_impl:1192 Starting.  Called from lib_teardown:182.
2023-07-18 08:51:36.091 thread(6) tag(0) SPEW cond_signal_impl:1208 Done for call at lib_teardown:182.
2023-07-18 08:51:36.091 thread(6) tag(0) INFO lib_teardown:186 Tearing down tag tickler thread.
2023-07-18 08:51:36.091 thread(6) tag(0) INFO lib_teardown:193 Tearing down tag tickler condition var.
...
2023-07-18 08:51:36.091 thread(6) tag(0) INFO lib_teardown:212 Done.

The tag_tickler_func thread is signaled by cond_signal_impl but never responds in the log. No cond_wait_impl call and no termination. As if the thread is already gone. But Windows 10 just moves past cond_signal_impl and tears down the library.

When running the sample on Windows 7 with the preemtive Shutdown() commented it looks like this:

2023-07-18 08:51:36.007 thread(2) tag(0) SPEW cond_wait_impl:1125 Starting. Called from tag_tickler_func:634.
...
2023-07-18 08:51:36.091 thread(6) tag(0) INFO plc_tag_shutdown:1095 Starting.
...
2023-07-18 08:51:36.091 thread(6) tag(0) INFO lib_teardown:176 Tearing down library.
...
2023-07-18 08:51:36.091 thread(6) tag(0) INFO lib_teardown:181 Signaling tag tickler condition var.
2023-07-18 08:51:36.091 thread(6) tag(0) SPEW cond_signal_impl:1192 Starting.  Called from lib_teardown:182

The tag_tickler_func thread is signaled by cond_signal_impl but never responds in the log. But there the main thread is stuck.
I tried to analyze this with Process Explorer and got a single remaining thread stuck with a call stack:

My workaround is to preemtively call Shutdown().

timyhac · Answer 1 · Wed Jul 19 2023 18:20:56 GMT+0800 (China Standard Time)

Hi @Teldaril - thanks for the message, and welcome to the libplctag community.

Is it a problem that you need to call Shutdown when exiting the program?

Please note that you need to dispose of the tag before calling Shutdown.
https://github.com/libplctag/libplctag.NET/blob/master/src/libplctag/LibPlcTag.cs#L108C85-L108C85

P.S. do the remarks in the documentation comments correlate with what you're seeing?

Teldaril · Answer 2 · Wed Jul 19 2023 19:27:26 GMT+0800 (China Standard Time)

Thanks for the fast answer!

It is not really a problem to call it preemtively. But it is not as clean as i would like it, because it will get called again when the plctag.dll unloads. It won't do anything, because all ressources are already cleared, but it is kind of a redundant call.
The log output is the same when i dispose the tag before the shutdown. I don't know if it is still absolutely necessary because the shutdown in the plctag.dll already clears the tags itself:
https://github.com/libplctag/libplctag/blob/release/src/lib/lib.c#L1091
Looks like it was added with the latest release:
libplctag/libplctag@v2.5.4...v2.5.5#diff-abcaf305164b51237221bfc1038729b50ac43e0eb88a40e18e109b8f33e33b7eR1091
They correlate in the sense that the shutdown function of the plctag.dll is called when the lib gets unloaded. But on Windows 7 it gets stuck at the unload handling as described above.

Kyle Hayes · Answer 3 · Wed Jul 19 2023 22:15:42 GMT+0800 (China Standard Time)

The plc_tag_shutdown() function was added to help with LabVIEW. For some reason, LabVIEW did not completely unload the DLL correctly without that. In normal use, just closing the last tag handle will cause the library to clean up all internal resources. On Linux and macOS, it is not needed at all.

After closing the last tag, calling plc_tag_shutdown() should be redundant, but harmless. It probably is a good idea to call it if you are going to explicitly (or implicitly) unload the DLL.

I do not have a running Windows 7 VM anymore so duplicating your results may be challenging :-(

Kyle Hayes · Answer 4 · Wed Jul 19 2023 22:23:59 GMT+0800 (China Standard Time)

If one of the threads is hanging, that might explain why the wake call is not working. Windows is a little more challenging to figure out as the wake model does not fit it quite as well as POSIX-based OSes like Linux and macOS.

Thanks for the debugging output. SPEW (named appropriately) really does dump nearly everything that goes on, so maybe there is a hint somewhere in there. Can you attach a full log of the output?

Teldaril · Answer 5 · Thu Jul 20 2023 15:00:32 GMT+0800 (China Standard Time)

Hello,

i have logs for 8 scenarios:

var tag = new Tag<IntPlcMapper, short>()
{
    Name = "N43:21",
    Gateway = "0.0.0.0",
    PlcType = PlcType.Slc500,
    Protocol = Protocol.ab_eip,
    Timeout = TimeSpan.FromSeconds(1),
    DebugLevel = DebugLevel.Spew
};

try
{
    tag.Read();
}
catch (Exception e)
{
    Console.WriteLine(e);
}

Thread.Sleep(1000);

tag.Dispose();
LibPlcTag.Shutdown();

return 0;

On Win10:
Win10_DisposeTagAndLib.txt
On Win7:
Win7_DisposeTagAndLib.txt

var tag = new Tag<IntPlcMapper, short>()
{
    Name = "N43:21",
    Gateway = "0.0.0.0",
    PlcType = PlcType.Slc500,
    Protocol = Protocol.ab_eip,
    Timeout = TimeSpan.FromSeconds(1),
    DebugLevel = DebugLevel.Spew
};

try
{
    tag.Read();
}
catch (Exception e)
{
    Console.WriteLine(e);
}

Thread.Sleep(1000);

//tag.Dispose();
LibPlcTag.Shutdown();

return 0;

On Win10:
Win10_DisposeLib.txt
On Win7:
Win7_DisposeLib.txt

var tag = new Tag<IntPlcMapper, short>()
{
    Name = "N43:21",
    Gateway = "0.0.0.0",
    PlcType = PlcType.Slc500,
    Protocol = Protocol.ab_eip,
    Timeout = TimeSpan.FromSeconds(1),
    DebugLevel = DebugLevel.Spew
};

try
{
    tag.Read();
}
catch (Exception e)
{
    Console.WriteLine(e);
}

Thread.Sleep(1000);

tag.Dispose();
//LibPlcTag.Shutdown();

return 0;

On Win10:
Win10_DisposeTag.txt
On Win 7 (stuck unloading plctag.dll):
Win7_DisposeTag.txt

var tag = new Tag<IntPlcMapper, short>()
{
    Name = "N43:21",
    Gateway = "0.0.0.0",
    PlcType = PlcType.Slc500,
    Protocol = Protocol.ab_eip,
    Timeout = TimeSpan.FromSeconds(1),
    DebugLevel = DebugLevel.Spew
};

try
{
    tag.Read();
}
catch (Exception e)
{
    Console.WriteLine(e);
}

Thread.Sleep(1000);

//tag.Dispose();
//LibPlcTag.Shutdown();

return 0;

On Win10:
Win10_DisposeNothing.txt
On Win7 (stuck unloading plctag.dll):
Win7_DisposeNothing.txt

Additionally the only thread left when unloading and stuck on Win7 is the main thread with the stack trace shown in my initial post. For me it looks like the tag_tickler_func thread is killed before the shutdown wants to handle this thread and the WakeConditionVariable in the shutdown is stuck for some reason.