STATUS_ENTRYPOINT_NOT_FOUND error with x86_64-pc-windows-gnu on Windows Server 2019 in GitHub action

[patowen]

The following GitHub action fails with nightly-2024-02-27 but passes with nightly-2024-02-26 for (I believe) any project with a build.rs file:

name: ci
on:
  workflow_dispatch:
  pull_request:
  push:
    branches:
    - main
  schedule:
  - cron: '00 01 * * *'
jobs:
  build:
    name: build

    env:
      # Emit backtraces on panics.
      RUST_BACKTRACE: 1
      CARGO_INCREMENTAL: 0

    runs-on: windows-2019
    strategy:
      fail-fast: false
      matrix:
        rust_version:
          - nightly-2024-02-26
          - nightly-2024-02-27
          - nightly-2024-02-28

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Install Rust
      uses: actions-rs/toolchain@v1
      with:
        toolchain: ${{ matrix.rust_version }}-x86_64-pc-windows-gnu
        target: x86_64-pc-windows-gnu
        profile: minimal
        override: true

    - name: Build
      shell: bash
      run: cargo build --verbose --target x86_64-pc-windows-gnu

The error I got is the following:

Run cargo build --verbose --target x86_64-pc-windows-gnu
   Compiling proc-macro2-testing-for-lavagna v0.1.0 (D:\a\lavagna\lavagna)
     Running `rustc --crate-name build_script_build --edition=2021 build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=b191affa0bd3624f -C extra-filename=-b191affa0bd3624f --out-dir 'D:\a\lavagna\lavagna\target\debug\build\proc-macro2-testing-for-lavagna-b191affa0bd3624f' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps'`
     Running `D:\a\lavagna\lavagna\target\debug\build\proc-macro2-testing-for-lavagna-b191affa0bd3624f\build-script-build`
error: failed to run custom build command for `proc-macro2-testing-for-lavagna v0.1.0 (D:\a\lavagna\lavagna)`

Caused by:
  process didn't exit successfully: `D:\a\lavagna\lavagna\target\debug\build\proc-macro2-testing-for-lavagna-b191affa0bd3624f\build-script-build` (exit code: 0xc0000139, STATUS_ENTRYPOINT_NOT_FOUND)
Error: Process completed with exit code 101.

This only affects x86_64-pc-windows-gnu (not MVSC), and switching from windows-2019 to windows-2022 fixes the issue.

I have reproduced the issue in patowen/lavagna#1. I may end up deleting that repo in the future, so for posterity, I've included all the code below:

Example project to reproduce the issue:

.github\workflows\ci.yml:

(See above)

src\main.rs:

fn main() {
    println!("Hello world");
}

build.rs:

fn main() {
    println!("I am in a build script.");
}

Cargo.toml:

[package]
name = "github-action-testing-for-lavagna"
version = "0.1.0"
edition = "2021"

The original discovery of this issue is reported in alepez/lavagna#29.

Meta

Running rustc --version --verbose yielded the following for the latest nightly version without the issue:

rustc 1.78.0-nightly (0ecbd0605 2024-02-25)
binary: rustc
commit-hash: 0ecbd0605770f45c9151715e66ba2b3cae367fcb
commit-date: 2024-02-25
host: x86_64-pc-windows-gnu
release: 1.78.0-nightly
LLVM version: 18.1.0

Running rustc --version --verbose on three versions with the issue (the next nightly version, the latest nightly version, and the current beta) yielded the following:

rustc 1.78.0-nightly (fc3800f65 2024-02-26)
binary: rustc
commit-hash: fc3800f65777a365b5125706d60f97e4d0675efe
commit-date: 2024-02-26
host: x86_64-pc-windows-gnu
release: 1.78.0-nightly
LLVM version: 18.1.0

rustc 1.79.0-nightly (ccfcd950b 2024-04-15)
binary: rustc
commit-hash: ccfcd950b333fed046275dd8d54fe736ca498aa7
commit-date: 2024-04-15
host: x86_64-pc-windows-gnu
release: 1.79.0-nightly
LLVM version: 18.1.3

rustc 1.78.0-beta.7 (6fd191292 2024-04-12)
binary: rustc
commit-hash: 6fd1912922a7b468b1875d16571bb7a00d5df93f
commit-date: 2024-04-12
host: x86_64-pc-windows-gnu
release: 1.78.0-beta.7
LLVM version: 18.1.2

This issue is not present in rustc 1.77.2 (the latest stable version).

[ChrisDenton]

The imports should mostly be shared between msvc and gnu so the fact the targets differ is a bit puzzling. My hypothesis would be that mingw on Windows Server 2019 is using an older version of an import library that maps one or more functions to the wrong DLL. Specifically I think this could be caused by the libsynchronization.a import library.

cc @mati865 in case you've heard of anything like this before

[mati865]

I doubt it's a problem with the old MinGW-w64; in that case, the program would fail to link.

I helped with a similar issue recently, and it was caused by the wrong comctl32.dll being loaded. There are two DLLs called that, and if a manifest is not provided, Windows will load the wrong one: zed-industries/zed#9757

[ChrisDenton]

Hm... I ran dumpbin /imports on both windows-2019 and windows-latest and the import tables do look different (see test repo).

windows-2019

Section contains the following imports:

    KERNEL32.dll
                4D0654 Import Address Table
                4D008C Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          14 AddVectoredExceptionHandler
                          6E CancelIo
                          83 CloseHandle
                          96 CompareStringOrdinal
                          A8 CopyFileExW
                          B8 CreateDirectoryW
                          BC CreateEventW
                          C1 CreateFileMappingA
                          C8 CreateFileW
                          CC CreateHardLinkW
                          D9 CreateNamedPipeW
                          E2 CreateProcessW
                          EC CreateSymbolicLinkW
                          EE CreateThread
                         34E HeapReAlloc
                         35A InitOnceBeginInitialize
                         35B InitOnceComplete
                         360 InitializeCriticalSection
                         364 InitializeProcThreadAttributeList
                         3B8 LeaveCriticalSection
                         3D7 MapViewOfFile
                         3DC Module32FirstW
                         3DE Module32NextW
                         3E1 MoveFileExW
                         3E8 MultiByteToWideChar
                         446 QueryPerformanceCounter
                         447 QueryPerformanceFrequency
                         459 RaiseException
                         467 ReadConsoleW
                         469 ReadFile
                         46A ReadFileEx
                         488 RemoveDirectoryW
                         49C RtlAddFunctionTable
                         49D RtlCaptureContext
                         4A4 RtlLookupFunctionEntry
                         4AA RtlUnwindEx
                         4AB RtlVirtualUnwind
                         4E3 SetCurrentDirectoryW
                         4EE SetEnvironmentVariableW
                         4F8 SetFileAttributesW
                         4FB SetFileInformationByHandle
                         4FE SetFilePointerEx
                         501 SetFileTime
                         508 SetHandleInformation
                         50B SetLastError
                         536 SetThreadStackGuarantee
                         543 SetUnhandledExceptionFilter
                         54B SetWaitableTimer
                         551 Sleep
                         554 SleepEx
                         55B SwitchToThread
                         560 TerminateProcess
                         572 TlsAlloc
                         573 TlsFree
                         574 TlsGetValue
                         575 TlsSetValue
                         582 UnhandledExceptionFilter
                         585 UnmapViewOfFile
                         58F UpdateProcThreadAttribute
                         5A4 VirtualProtect
                         5A6 VirtualQuery
                         5AC WaitForMultipleObjects
                         5AE WaitForSingleObject
                         5B6 WaitOnAddress
                         5B8 WakeByAddressAll
                         5B9 WakeByAddressSingle
                         5D3 WideCharToMultiByte
                         5E6 WriteConsoleW
                         5E8 WriteFileEx
                         5F8 __C_specific_handler

    msvcrt.dll
                4D0A1C Import Address Table
                4D0454 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          52 __getmainargs
                          53 __initenv
                          54 __iob_func
                          5B __lconv_init
                          61 __set_app_type
                          63 __setusermatherr
                          73 _acmdln
                          7B _amsg_exit
                          8D _cexit
                          FD _fmode
                         101 _fpreset
                         14B _initterm
                         263 _onexit
                         407 abort
                         41A calloc
                         427 exit
                         43A fprintf
                         441 free
                         44D fwrite
                         47C malloc
                         4A2 signal
                         4BA strncmp
                         4DA vfprintf

    ntdll.dll
                4D0ADC Import Address Table
                4D0514 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                         107 NtCreateFile
                         1D7 NtReadFile
                         254 NtWriteFile
                         43D RtlNtStatusToDosError
                         7A9 memcmp
                         7AA memcpy
                         7AC memmove
                         7AE memset
                         7BE strlen

    USERENV.dll
                4D0B2C Import Address Table
                4D0564 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          24 GetUserProfileDirectoryW

    WS2_32.dll
                4D0B3C Import Address Table
                4D0574 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          1F WSACleanup
                          27 WSADuplicateSocketW
                          30 WSAGetLastError
                          49 WSARecv
                          4E WSASend
                          58 WSASocketW
                          59 WSAStartup
                          A2 accept
                          A3 bind
                          A4 closesocket
                          A5 connect
                          A6 freeaddrinfo
                          A7 getaddrinfo
                          AC getpeername
                          B1 getsockname
                          B2 getsockopt
                          B9 ioctlsocket
                          BA listen
                          BD recv
                          BE recvfrom
                          BF select
                          C0 send
                          C1 sendto
                          C2 setsockopt
                          C3 shutdown

    bcryptprimitives.dll
                4D0C0C Import Address Table
                4D0644 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                           1 ProcessPrng

windows-latest

Section contains the following imports:

    KERNEL32.dll
             1400D0764 Import Address Table
             1400D012C Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          14 AddVectoredExceptionHandler
                          78 CancelIo
                          8D CloseHandle
                          A1 CompareStringOrdinal
                          B3 CopyFileExW
                          C3 CreateDirectoryW
                          C8 CreateEventW
                          CD CreateFileMappingA
                          D4 CreateFileW
                          D8 CreateHardLinkW
                          E5 CreateNamedPipeW
                          ED CreateProcessW
                          F8 CreateSymbolicLinkW
                          FA CreateThread
                         103 CreateToolhelp32Snapshot
                         108 CreateWaitableTimerExW
                         119 DeleteCriticalSection
                         11E DeleteFileW
                         11F DeleteProcThreadAttributeList
                         129 DeviceIoControl
                         137 DuplicateHandle
                         13D EnterCriticalSection
                         16C ExitProcess
                         183 FindClose
                         18E FindFirstFileW
                         19A FindNextFileW
                         1AD FlushFileBuffers
                         1B5 FormatMessageW
                         1B8 FreeEnvironmentStringsW
                         1E6 GetCommandLineW
                         20B GetConsoleMode
                         220 GetCurrentDirectoryW
                         226 GetCurrentProcess
                         227 GetCurrentProcessId
                         22A GetCurrentThread
                         249 GetEnvironmentStringsW
                         24B GetEnvironmentVariableW
                         24E GetExitCodeProcess
                         259 GetFileAttributesW
                         25B GetFileInformationByHandle
                         25C GetFileInformationByHandleEx
                         262 GetFileType
                         264 GetFinalPathNameByHandleW
                         26D GetFullPathNameW
                         274 GetLastError
                         288 GetModuleFileNameW
                         289 GetModuleHandleA
                         28C GetModuleHandleW
                         2AE GetOverlappedResult
                         2C4 GetProcAddress
                         2CA GetProcessHeap
                         2CC GetProcessId
                         2E8 GetStdHandle
                         2F6 GetSystemDirectoryW
                         2F9 GetSystemInfo
                         300 GetSystemTimePreciseAsFileTime
                         30C GetTempPathW
                         33E GetWindowsDirectoryW
                         35D HeapAlloc
                         363 HeapFree
                         367 HeapReAlloc
                         373 InitOnceBeginInitialize
                         374 InitOnceComplete
                         37A InitializeCriticalSection
                         37F InitializeProcThreadAttributeList
                         3D6 LeaveCriticalSection
                         3F9 MapViewOfFile
                         3FE Module32FirstW
                         400 Module32NextW
                         403 MoveFileExW
                         40A MultiByteToWideChar
                         469 QueryPerformanceCounter
                         46A QueryPerformanceFrequency
                         47F RaiseException
                         48D ReadConsoleW
                         490 ReadFile
                         491 ReadFileEx
                         4AF RemoveDirectoryW
                         4C5 RtlCaptureContext
                         4CC RtlLookupFunctionEntry
                         4D2 RtlUnwindEx
                         4D3 RtlVirtualUnwind
                         50B SetCurrentDirectoryW
                         516 SetEnvironmentVariableW
                         520 SetFileAttributesW
                         523 SetFileInformationByHandle
                         526 SetFilePointerEx
                         529 SetFileTime
                         530 SetHandleInformation
                         534 SetLastError
                         563 SetThreadStackGuarantee
                         56F SetUnhandledExceptionFilter
                         578 SetWaitableTimer
                         57F Sleep
                         582 SleepEx
                         589 SwitchToThread
                         58E TerminateProcess
                         5A0 TlsAlloc
                         5A1 TlsFree
                         5A2 TlsGetValue
                         5A3 TlsSetValue
                         5B3 UnmapViewOfFile
                         5BD UpdateProcThreadAttribute
                         5D1 VirtualProtect
                         5D3 VirtualQuery
                         5DA WaitForMultipleObjects
                         5DC WaitForSingleObject
                         608 WideCharToMultiByte
                         61B WriteConsoleW
                         61D WriteFileEx
                         62D __C_specific_handler

    api-ms-win-crt-environment-l1-1-0.dll
             1400D0AE4 Import Address Table
             1400D04AC Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                           1 __p__environ
                           2 __p__wenviron

    api-ms-win-crt-heap-l1-1-0.dll
             1400D0AFC Import Address Table
             1400D04C4 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          18 _set_new_mode
                          19 calloc
                          1A free
                          1B malloc

    api-ms-win-crt-math-l1-1-0.dll
             1400D0B24 Import Address Table
             1400D04EC Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                           A __setusermatherr

    api-ms-win-crt-private-l1-1-0.dll
             1400D0B34 Import Address Table
             1400D04FC Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                         477 memcmp
                         478 memcpy
                         479 memmove

    api-ms-win-crt-runtime-l1-1-0.dll
             1400D0B54 Import Address Table
             1400D051C Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                           5 __p___argc
                           6 __p___argv
                           7 __p___wargv
                          17 _cexit
                          19 _configure_narrow_argv
                          1A _configure_wide_argv
                          1E _crt_at_quick_exit
                          1F _crt_atexit
                          27 _fpreset
                          36 _initialize_narrow_environment
                          38 _initialize_wide_environment
                          39 _initterm
                          45 _set_app_type
                          4B _set_invalid_parameter_handler
                          58 abort
                          59 exit
                          67 signal

    api-ms-win-crt-stdio-l1-1-0.dll
             1400D0BE4 Import Address Table
             1400D05AC Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                           1 __acrt_iob_func
                           2 __p__commode
                           3 __p__fmode
                           4 __stdio_common_vfprintf
                           8 __stdio_common_vfwprintf
                          A9 fwrite

    api-ms-win-crt-string-l1-1-0.dll
             1400D0C1C Import Address Table
             1400D05E4 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          A0 memset
                          A8 strlen
                          AB strncmp

    api-ms-win-crt-time-l1-1-0.dll
             1400D0C3C Import Address Table
             1400D0604 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                           9 __daylight
                           B __timezone
                           C __tzname
                          3C _tzset

    ntdll.dll
             1400D0C64 Import Address Table
             1400D062C Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                         13E NtCreateFile
                         234 NtReadFile
                         2C3 NtWriteFile
                         541 RtlNtStatusToDosError

    api-ms-win-core-synch-l1-2-0.dll
             1400D0C8C Import Address Table
             1400D0654 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          33 WaitOnAddress
                          35 WakeByAddressAll
                          36 WakeByAddressSingle

    USERENV.dll
             1400D0CAC Import Address Table
             1400D0674 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          24 GetUserProfileDirectoryW

    WS2_32.dll
             1400D0CBC Import Address Table
             1400D0684 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                          20 WSACleanup
                          28 WSADuplicateSocketW
                          31 WSAGetLastError
                          4A WSARecv
                          4F WSASend
                          59 WSASocketW
                          5A WSAStartup
                          A3 accept
                          A4 bind
                          A5 closesocket
                          A6 connect
                          A7 freeaddrinfo
                          A8 getaddrinfo
                          AD getpeername
                          B2 getsockname
                          B3 getsockopt
                          BA ioctlsocket
                          BB listen
                          BE recv
                          BF recvfrom
                          C0 select
                          C1 send
                          C2 sendto
                          C3 setsockopt
                          C4 shutdown

    bcryptprimitives.dll
             1400D0D8C Import Address Table
             1400D0754 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                           1 ProcessPrng

Note that WaitOnAddress is listed under KERNEL32 in windows-2019 but it's more correctly listed under api-ms-win-core-synch-l1-2-0.dll in windows-latest.

[mati865]

Yeah, that is probably the culprit but mingw-w64 links WaitOnAddress from api-ms-win-core-synch-l1-2-0.dll since mid 2016: mirror/mingw-w64@984233c

If it's caused by miingw-w64 toolchain preinstalled on windows-2019 image it must be terribly out of date.

[ChrisDenton]

Quite possibly it is that out of date. The other explanation would be it's coming from somewhere else, which Is a bit worrying.

[patowen]

Based on https://github.com/actions/runner-images/blob/main/images/windows/Windows2019-Readme.md, my first impression is that mingw-w64 isn't preinstalled on the windows-2019 image. Unless Windows Subsystem for Linux is relevant, I assume that the mingw-w64 toolchain is installed as part of the run of actions-rs/toolchain@v1.

[mati865]

Various tools like Perl might come with GCC. Could you determine location of gcc.exe and x86_64-w64-mingw32-gcc.exe?

[patowen]

I'm not sure. I've only been able to reproduce this via a GitHub action, not on my own machine, although I might be able to use something like https://github.com/mxschmitt/action-tmate to explore around. Alternatively, I could just run a fixed list of commands.

If there's a set of bash commands I can run to answer any questions you have, please let me know. I'm not sure how the Rust toolchain determines which version of gcc to run when building Rust code, but that seems like something you might know.

EDIT: Based on your question, I might need to just run which with the two applications you asked about.

EDIT2: gcc and x86_64-w64-mingw32-gcc are both in /c/mingw64/bin/ according to bash.

[patowen]

Here are the versions of gcc based on running --version:

gcc.exe (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

x86_64-w64-mingw32-gcc.exe (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[mati865]

Thanks, apparently there was a bug in mingw-w64 fixed in 2020: mirror/mingw-w64@68d0e7d
IIRC this toolchain is older than that so it's indeed a bug in mingw-w64 shipped with this windows-2019.

[patowen]

Oh, weird. windows-2019 also comes shipped with

rustc 1.77.1 (7cf61ebde 2024-03-27)
binary: rustc
commit-hash: 7cf61ebde7b22796c69757901dd346d0fe70bd97
commit-date: 2024-03-27
host: x86_64-pc-windows-msvc
release: 1.77.1
LLVM version: 17.0.6

so this image generally has up-to-date packages. I also didn't see mingw or mingw-w64 mentioned anywhere in its README, so now I'm wondering if there's a bug in the image itself.

I expect this means that no action will be needed for this particular issue unless we want to be more robust to bad environments like this.

EDIT: It looks like it's defined here: https://github.com/actions/runner-images/blob/7993ee591358052f1dd23c6135281e24b80b572f/images/windows/scripts/build/Install-Mingw64.ps1

EDIT2: It was documented after all. I had to search for "gcc", not "mingw".

[patowen]

I was rather confused because https://www.mingw-w64.org/downloads/ states that the latest version from the 8.x series is 8.0.2, but the version installed of mingw-builds in the the Windows 2019 image is 8.1.0, which seems like it shouldn't exist. However, based on the releases of https://github.com/niXman/mingw-builds-binaries, the number after the rt_v is probably what's important, which means that we're on version 6.0.0, which is indeed before the fix.

I might ask in the actions/runner-images project to see if it's worth updating mingw64 for this reason.

EDIT: Filed actions/runner-images#9702. Depending on their response, if they decide not to update, it's possible actions-rs/toolchain could be fixed instead to stop trying to install Rust over an incompatible version of MinGW.

[patowen]

For some reason, my original issue description code snippet mentioned windows-2022 instead of windows-2019. I think that's because I copy/pasted the wrong version, when I was seeing whether the windows-2022 workaround worked (which I believe it did). I apologize if that caused any confusion. I've now edited the description to fix this mistake.

[patowen]

Given that #124019 seems to be in progress and would close this issue, I mentioned that in actions/runner-images#9702 for extra transparency.

[patowen]

Thank you for all your help on this! It's awesome that all this happened in only 4 days.

I can confirm on my end that this issue is now fully resolved on the latest nightly release of Rust (2024-04-19).