For haskell/test-framework, HLS has mysterious errors showing up in VSCode

[andreasabel]

To reproduce:

gh repo clone haskell/test-framework
cd test-framework
gh pr checkout 75
haskell-language-server-9.10.1

This fails, while e.g. cabal build all -w ghc-9.10.1 succeeds without error.

The cabal.project here has:

packages: ./core
          ./hunit
          ./quickcheck2
          ./example

However, there is also a directory ./quickcheck containing legacy code for the deprecated package test-framework-quickcheck. This code is not functioning and should be ignored by a build process.

The first error by HLS is:

2025-02-24T21:53:32.808035Z | Info | Making new HscEnv. In-place unit ids: [ test-framework-0.8.2.1-inplace
                                      , test-framework-0.8.2.1-inplace-test-framework-tests
                                      , test-framework-example-0.2.1-inplace-test-framework-example
                                      , test-framework-hunit-0.3.0.2-inplace
                                      , test-framework-quickcheck2-0.3.0.6-inplace ]
2025-02-24T21:53:32.901458Z | Info | Cradle path: quickcheck/Test/Framework/Providers/QuickCheck.hs
2025-02-24T21:53:32.901530Z | Warning | No [cradle](https://github.com/mpickering/hie-bios#hie-bios) found for quickcheck/Test/Framework/Providers/QuickCheck.hs.
Proceeding with [implicit cradle](https://hackage.haskell.org/package/implicit-hie).
You should ignore this message, unless you see a 'Multi Cradle: No prefixes matched' error.
2025-02-24T21:53:32.902256Z | Info | invoking build tool to determine build flags (this may take some time depending on the cache)
2025-02-24T21:53:33.381584Z | Info | Load cabal cradle using single file
2025-02-24T21:53:33.768656Z | Info | cabal --builddir=/Users/abela/.cache/hie-bios/dist-test-framework-3a7e3fed834d796685ee68f42b350f7b v2-repl --with-compiler /Users/abela/.cache/hie-bios/wrapper-b54f81dea4c0e6d1626911c526bc4e36 --with-hc-pkg /Users/abela/.cache/hie-bios/ghc-pkg-b5af9e5b00fb1eb99fb148381418ddd7 /Users/abel/bin/src/test-framework/quickcheck/Test/Framework/Providers/QuickCheck.hs
  Environment Variables
    HIE_BIOS_OUTPUT: /private/var/folders/19/d9jtc4c5365g3c_5jjk7m_980000gn/T/HIE_BIOS_OUTPUT46741-6
    HIE_BIOS_GHC: /Users/abela/.ghcup/ghc/9.10.1/lib/ghc-9.10.1/bin/ghc-9.10.1
    HIE_BIOS_GHC_ARGS: -B/Users/abela/.ghcup/ghc/9.10.1/lib/ghc-9.10.1/lib
2025-02-24T21:53:33.797389Z | Info | updateFileDiagnostics published different from new diagnostics - file diagnostics: File:     /Users/abel/bin/src/test-framework/quickcheck/Test/Framework/Providers/QuickCheck.hs
Hidden:   no
Range:    1:1-2:1
Source:   cradle
Severity: DiagnosticSeverity_Error
Message: 
  Loading the module
  '/Users/abel/bin/src/test-framework/quickcheck/Test/Framework/Providers/QuickCheck.hs' failed.

  It may not be listed in your .cabal file!
  Perhaps you need to add `QuickCheck` to other-modules or exposed-modules.

  For more information, visit:
  https://cabal.readthedocs.io/en/3.4/developing-packages.html#modules-included-in-the-package

HLS seems to leave the path to salvation here, by getting seduced to look into the broken package:

No cradle found for quickcheck/Test/Framework/Providers/QuickCheck.hs.
Proceeding with implicit cradle.

From there the road to hell is paved:

Loading the module
'/Users/abel/bin/src/test-framework/quickcheck/Test/Framework/Providers/QuickCheck.hs' failed.
It may not be listed in your .cabal file!

One wants to happily ignore this error, but it seems to derail HLS because we later see:

Severity: DiagnosticSeverity_Error
Message:
attempting to use module ‘test-framework-0.8.2.1:Test.Framework.Utilities’
(/Users/abel/bin/src/test-framework/core/src/Test/Framework/Utilities.hs) which is not loaded

This then causes a real type error:

Severity: DiagnosticSeverity_Error
Message: Variable not in scope: microsecondsToPicoseconds :: t0 -> Integer

Maybe HLS should not poke its nose into packages not listed in cabal.project or stack.yaml...

[andreasabel]

Severity: DiagnosticSeverity_Error
Message:
attempting to use module ‘test-framework-0.8.2.1:Test.Framework.Utilities’
(/Users/abel/bin/src/test-framework/core/src/Test/Framework/Utilities.hs) which is not loaded

Actually, this error is even appearing if the irrelevant directory quickcheck/ is deleted.
So while there is some argument that maybe HLS should ignore sub-projects not included in cabal.project or stack.yaml, this does not seem to be the cause of the confusion.

It may be that the particular project structure is not handled correctly by HLS (or its underlying tools). I am seeing:

• attempting to use module ‘test-framework-0.8.2.1:Test.Framework.Utilities’
  (test-framework/core/src/Test/Framework/Utilities.hs) which is not loaded

• A warning that the import of this module is redundant
• Scope errors for symbols that should actually have been imported from this module (as its import is factually not redundant)

[fendor]

Am I right to assume, this only happens when running HLS on the cli?

This issue is known, the HLS cli is not well-developed. HLS just traverses the current directory for all haskell files it can find (often including dist-newstyle and other generated files) and tries to load them.

We could argue, HLS should honour any potential hie.yaml configuration which specifies a none-cradle. I think that should work? See #3270
However, neither stack nor cabal give us any information about the Haskell files in a package upfront, so we have no choice but to ask for each file whether we can load them. In this case, that just fails for Haskell files that should be ignored.
I don't know whether there is anything we can do, except for going the long route and trying to fix this in stack and cabal.

[andreasabel]

Am I right to assume, this only happens when running HLS on the cli?

Originally, I noticed problems in VSCode, but I thought running the HLS cli would be the way to report bugs, or did I misunderstand this?

[fendor]

Unfortunately, it depends on the kind of issue you are reporting :) Since the CLI and LSP mode are not always identical. Generally, both ways are fine and appreciated to report issues, it just so happens that I made some assumptions about this issue based on how it is reported. Which might be my mistake.
I understood your report to be related to the cli mode, that the cli mode shows you diagnostics (wrong ones) for files that are not part of the cabal project. Which I think is hard to avoid.
Do you mean, you get diagnostics in the editor when you are opening Haskell files that are not part of the cabal/stack project?
Or even get these diagnostics when you are not opening them in your editor directly?

[andreasabel]

Do you mean, you get diagnostics in the editor when you are opening Haskell files that are not part of the cabal/stack project?

In this case, I get the correct error that files are not belonging to the project.

In my initial report, I thought that the original reason for the malfunctioning in VSCode was that HLS considered also directories outside the project.
I'd like to take that back, and I tried so in my subsequent comment.
The thing with the untracked directory isn't really the problem that concerns me, and it might just be a distraction.
Should I start from scratch and open a new issue with the distraction removed?

So the real issue is that if I open the test-framework project in VSCode and click around on the source files, I get at some point strange errors that do not appear with cabal build all or stack build.
These errors prevent the HLS to provide the services I want, e.g. the feature where you can make imports lists explicit by just a click.

I attach screenshots this time (to not divert the focus to the CLI version of HLS). However, these errors are also reported by the CLI.

I suppose the problems are reproducible by cloning https://github.com/haskell/test-framework , opening the root in VSCode and randomly open files from its subprojects and subfolders (even excluding the quickcheck folder that is not tracked in cabal.project).