Cannot import package from a private repo inside a group

[ddobrinskiy]

I am trying to import a module from a private repo.

import "gitlab.private-repo.com/sre/scout/flowgraph-api.git"

Notice that plowgraph-api lives inside scout group inside sre , which may complicate things.

Go get from command line outside jupyter works:

❯ go get -v gitlab.private-repo.com/sre/scout/flowgraph-api.git
get "gitlab.private-repo.com/sre/scout": found meta tag get.metaImport{Prefix:"gitlab.private-repo.com/sre/scout", VCS:"git", RepoRoot:"https://gitlab.private-repo.com/sre/scout.git"} at //gitlab.private-repo.com/sre/scout?go-get=1
go get: gitlab.private-repo.com/sre/scout/flowgraph-api.git@v1.3.2: parsing go.mod:
        module declares its path as: gitlab.private-repo.com/sre/scout/flowgraph-api
                but was required as: gitlab.private-repo.com/sre/scout/flowgraph-api.git

However importing in jupyter, I get the following error
error loading package "gitlab.private-repo.com/sre/scout/flowgraph-api.git/api/client" metadata: packages.Load() could not find package "gitlab.private-repo.com/sre/scout/flowgraph-api.git/api/client"

I tried multiple formatts for import statements

❯ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/david/.cache/go-build"
GOENV="/home/david/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/david/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/home/david/go/go1.13.10/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/home/david/go/go1.13.10/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build040957544=/tmp/go-build -gno-record-gcc-switches"

Any help would be appreciated, thanks!

[cosmos72]

Could you post the output of the following two commands?

GO111MODULES=on go get -v gitlab.private-repo.com/sre/scout/flowgraph-api.git
GO111MODULES=on go list -v gitlab.private-repo.com/sre/scout/flowgraph-api.git

Being a private repository, I do not have a simple way to reproduce your configuration.

[ddobrinskiy]

@cosmos72 Terribly sorry for the inconvenience. Unfortunately, the repo contains proprietary code, so I can't make it public.

Please find the requested output below.

Note that there is no repo "https://gitlab.private-repo.com/sre/scout.git", scout is a gitlab group containing other repos (flowgraph-api being one of them)

P.s. do I understand correctly that these commands should be run from my terminal, and not from within jupyter?

❯ GO111MODULES=on go get -v gitlab.private-repo.com/sre/scout/flowgraph-api.git
get "gitlab.private-repo.com/sre/scout": found meta tag get.metaImport{Prefix:"gitlab.private-repo.com/sre/scout", VCS:"git", RepoRoot:"https://gitlab.private-repo.com/sre/scout.git"} at //gitlab.private-repo.com/sre/scout?go-get=1
go get: gitlab.private-repo.com/sre/scout/flowgraph-api.git@v1.3.2: parsing go.mod:
        module declares its path as: gitlab.private-repo.com/sre/scout/flowgraph-api
                but was required as: gitlab.private-repo.com/sre/scout/flowgraph-api.git
❯ GO111MODULES=on go list -v gitlab.private-repo.com/sre/scout/flowgraph-api.git
get "gitlab.private-repo.com/sre/scout": found meta tag get.metaImport{Prefix:"gitlab.private-repo.com/sre/scout", VCS:"git", RepoRoot:"https://gitlab.private-repo.com/sre/scout.git"} at //gitlab.private-repo.com/sre/scout?go-get=1
go: gitlab.private-repo.com/sre/scout/flowgraph-api.git: gitlab.private-repo.com/sre/scout/flowgraph-api.git@v1.3.2: parsing go.mod:
        module declares its path as: gitlab.private-repo.com/sre/scout/flowgraph-api
                but was required as: gitlab.private-repo.com/sre/scout/flowgraph-api.git

[cosmos72]

Yes, you understood correctly.
I hoped they would produce some error - possibly the same you get in Jupyter.
But alas they work, so I am running out of ideas on how to reproduce (and analyze) this issue.

What about the plain import "gitlab.private-repo.com/sre/scout/flowgraph-api" ?

[ddobrinskiy]

@cosmos72 thanks for trying!
I will try to use a public repo with grouping soon to make it more reproducible.

Meanwhile:

IN: import "gitlab.ozon.ru/sre/scout/flowgraph-api"

error loading package "gitlab.ozon.ru/sre/scout/flowgraph-api" metadata: -: git ls-remote -q https://gitlab.ozon.ru/sre/scout.git in /home/david/go/pkg/mod/cache/vcs/556ea6c16783008b47b9d2a6437eca9e3a8bda6b17e5983a60502b50217b372c: exit status 128:
	remote: 
	remote: ========================================================================
	remote: 
	remote: The project you were looking for could not be found.
	remote: 
	remote: ========================================================================
	remote: 
	fatal: Could not read from remote repository.
	
	Please make sure you have the correct access rights
	and the repository exists.

[ddobrinskiy]

@cosmos72 Hi, so I created a public repo with similar subgroup structure, by forking gonum. See it here:

https://gitlab.com/testgroup212/subgroup/gonum

However I'm having trouble figuring out how go handles module installation and what it expects.

Maybe you have an example of a minimal repo I could fork there and try? Looks like gonum has it's expected installation path hard-coded somewhere

❯ GO111MODULES=on go get -v gitlab.com/testgroup212/subgroup/gonum
get "gitlab.com/testgroup212/subgroup/gonum": found meta tag get.metaImport{Prefix:"gitlab.com/testgroup212/subgroup/gonum", VCS:"git", RepoRoot:"https://gitlab.com/testgroup212/subgroup/gonum.git"} at //gitlab.com/testgroup212/subgroup/gonum?go-get=1
gitlab.com/testgroup212/subgroup/gonum (download)
package gitlab.com/testgroup212/subgroup/gonum: code in directory /home/david/go/src/gitlab.com/testgroup212/subgroup/gonum expects import "gonum.org/v1/gonum"

[ddobrinskiy]

Sorry for spamming with multiple comments, but on an older note. Could you help me figure out where the following error message is generated:

error loading package "gitlab.private-repo.com/sre/scout/flowgraph-api.git/api/client" metadata: packages.Load() could not find package "gitlab.private-repo.com/sre/scout/flowgraph-api.git/api/client"

Specifically packages.Load() could not find package

[ddobrinskiy]

Grabbing at straws here.

Is there a way for gophernotes imports to be go.mod aware?

I need to use this go.mod for my code to work outside of jupyter

module gitlab.private-repo.com/sre/scout/analytics

go 1.13

require gitlab.private-repo.com/sre/scout/flowgraph-api v1.3.2

replace gitlab.private-repo.com/sre/scout/flowgraph-api => gitlab.private-repo.com/sre/scout/flowgraph-api.git v1.3.2

[cosmos72]

packages.Load() could not find package means the function golang.org/x/tools/go/packages.Load() returned an error. The package golang.org/x/tools/go is the fundamental library to work with Go modules.

About your second question: gophernotes imports create a go.mod in a temporary directory. There is currently no way to customize such generated go.mod.

I am trying to imagine what such a customization API could look like, and the lesser evil seems to be something like "append the content of a known file - for example $HOME/.gophernotes.go.mod - to the generated go.mod"

[ddobrinskiy]

Yes, it seems that for my case some modification on go.mod is required.

I'm new to go, so not sure how often a custom go.mod may be required. On the first glance, however, seems that it may be useful for jupyter to be used as a full-fetched IDE.

I am currently porting my Python code to Go, and doing so without jupyter is quite painful :)

p.s. as a very dirty hack, maybe I could try to overwrite the temporary go.mod file? Any suggestions?

[cosmos72]

The easiest workaround is to modify the code not to create the temporary go.mod file if it exists already. Then you can prepare your customized file and it will be used while importing.

To do that, you need to patch the function createPluginGoModFile() in package github.com/cosmos72/gomacro/base/genimport - since that's a dependency of gophernotes, you need to clone two github repositories, patch the function, and recompile gophernotes with GO111MODULE=off (the compiled binary will still support modules).

A slightly more detailed walkthrough is:

# download gomacro and gophernotes in GOPATH
GO111MODULE=off go get -v github.com/cosmos72/gomacro
GO111MODULE=off go get -v github.com/gopherdata/gophernotes

# modify the function createPluginGoModFile() with your favourite editor:
cd "`go env GOPATH`/src/github.com/cosmos72/gomacro/base/genimport"
$EDITOR importer.go 

# build and install  gomacro and gophernotes
cd "`go env GOPATH`/src/github.com/cosmos72/gomacro"
GO111MODULE=off go install -v -i
cd "`go env GOPATH`/src/github.com/gopherdata/gophernotes"
GO111MODULE=off go install -v -i