Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: API for self-hosted bitbucket instance #9134

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

oleg-andreyev
Copy link
Contributor

@oleg-andreyev oleg-andreyev commented Aug 18, 2020

By using Bitbucket Server API composer install "should" speed up a bit

Some testing results (could be affected by bitbucket-server configuration)

without changes + without cache + install

[9.0MiB/10.25s] Memory usage: 9.01MiB (peak: 9.32MiB), time: 10.25s
../../composer/bin/composer install --profile -vvv --no-cache  0.65s user 0.50s system 10% cpu 10.610 total
---
[8.9MiB/5.78s]   - Installing foobar/communication-lib (1.0.5): [8.9MiB/5.78s]
[8.9MiB/5.78s]     Cloning to cache at '/test-composer/./.cache/https---*******-scm-lib-communication-lib.git/'
[9.0MiB/5.78s] Executing command (CWD): git clone --mirror 'https://*******/scm/lib/communication-lib.git' '/test-composer/./.cache/https---*******-scm-lib-communication-lib.git/'
[9.0MiB/7.08s] Executing command (CWD): git clone --mirror 'https://user:***@*******/scm/lib/communication-lib.git' '/test-composer/./.cache/https---*******-scm-lib-communication-lib.git/'
[9.0MiB/9.76s] Executing command (/test-composer/./.cache/https---*******-scm-lib-communication-lib.git/): git rev-parse --git-dir
[9.0MiB/9.77s] Executing command (/test-composer/./.cache/https---*******-scm-lib-communication-lib.git/): git rev-parse --quiet --verify '42ed18843dfcf3313a33fc81a2e897b902f036b8^{commit}'
[9.0MiB/9.78s] Cloning 42ed18843dfcf3313a33fc81a2e897b902f036b8 from cache
[9.0MiB/9.78s] Executing command (CWD): git clone --no-checkout '/test-composer/./.cache/https---*******-scm-lib-communication-lib.git/' '/test-composer/vendor/foobar/communication-lib' --dissociate --reference '/test-composer/./.cache/https---*******-scm-lib-communication-lib.git/' && cd '/test-composer/vendor/foobar/communication-lib' && git remote set-url origin 'https://*******/scm/lib/communication-lib.git' && git remote add composer 'https://*******/scm/lib/communication-lib.git'
[9.0MiB/9.86s] Executing command (/test-composer/vendor/foobar/communication-lib): git branch -r
[9.0MiB/9.87s] Executing command (/test-composer/vendor/foobar/communication-lib): git checkout '1.0.5' --
[9.0MiB/9.89s] Executing command (/test-composer/vendor/foobar/communication-lib): git reset --hard '42ed18843dfcf3313a33fc81a2e897b902f036b8' --
[9.0MiB/9.91s]     REASON: Required by the root package: Install command rule (install foobar/communication-lib 1.0.5)

without changes + without cache + update (without composer.lock)

[179.8MiB/27.19s] Memory usage: 179.76MiB (peak: 195.7MiB), time: 27.19s
../../composer/bin/composer update --profile -vvv --no-cache  1.83s user 1.35s system 11% cpu 27.550 total
---
[8.4MiB/0.01s] Executing command (CWD): git clone --mirror 'ssh://git@***********/lib/communication-lib.git' '/test-composer/./.cache/ssh---git-***********-lib-communication-lib.git/'
[8.4MiB/2.96s] Executing command (/test-composer/./.cache/ssh---git-***********-lib-communication-lib.git/): git show-ref --tags --dereference
[8.4MiB/2.98s] Executing command (/test-composer/./.cache/ssh---git-***********-lib-communication-lib.git/): git branch --no-color --no-abbrev -v
[8.4MiB/2.99s] Executing command (/test-composer/./.cache/ssh---git-***********-lib-communication-lib.git/): git branch --no-color
[8.4MiB/3.01s] Executing command (/test-composer/./.cache/ssh---git-***********-lib-communication-lib.git/): git show 'master':'composer.json'
[8.4MiB/3.02s] Executing command (/test-composer/./.cache/ssh---git-***********-lib-communication-lib.git/): git -c log.showSignature=false log -1 --format=%at 'master'
...
[8.6MiB/4.18s] Reading composer.json of foobar/communication-lib (master)

with changes + without cache + install

[9.0MiB/8.47s] Memory usage: 9.01MiB (peak: 9.32MiB), time: 8.47s
../../composer/bin/composer install --profile -vvv --no-cache  0.44s user 0.32s system 8% cpu 8.812 total
---
[8.9MiB/6.02s]   - Installing foobar/communication-lib (1.0.5): [8.9MiB/6.02s] Using HTTP basic authentication with username "user"
[9.0MiB/6.02s] Downloading https://*******/rest/api/1.0/projects/lib/repos/communication-lib/archive?at=42ed18843dfcf3313a33fc81a2e897b902f036b8&format=zip
[9.0MiB/6.02s] [9.0MiB/7.61s] Downloading (100%)[9.0MiB/7.61s]                [8.9MiB/7.61s] B/7.61s]
[8.9MiB/7.61s]  Extracting archive[8.9MiB/7.61s] Executing command (CWD): unzip -qq  '/test-composer/vendor/foobar/communication-lib/23e15b80e72a24ac37fe5cad4312caa4' -d '/test-composer/vendor/composer/eeb953bd'
[8.9MiB/7.64s]     REASON: Required by the root package: Install command rule (install foobar/communication-lib 1.0.5)

with changes + without cache + update (without composer.lock)

[179.6MiB/64.95s] Memory usage: 179.62MiB (peak: 195.5MiB), time: 64.95s
../../composer/bin/composer update --profile -vvv --no-cache  2.20s user 0.64s system 4% cpu 1:05.32 total
---
[8.3MiB/0.01s] Using HTTP basic authentication with username "user"
[8.3MiB/0.01s] Downloading https://***********/rest/api/1.0/projects/lib/repos/communication-lib
[8.3MiB/0.64s] Downloading https://***********/rest/api/1.0/projects/lib/repos/communication-lib/branches/default
[8.3MiB/1.26s] Downloading https://***********/rest/api/1.0/projects/lib/repos/communication-lib/raw/composer.json?at=master
[8.3MiB/1.89s] Downloading https://***********/rest/api/1.0/projects/lib/repos/communication-lib/commits/master
[8.3MiB/2.52s] Downloading https://***********/rest/api/1.0/projects/lib/repos/communication-lib/tags?start=0&limit=100&orderBy=MODIFICATION
[8.3MiB/3.15s] Downloading https://***********/rest/api/1.0/projects/lib/repos/communication-lib/branches?start=0&limit=100&orderBy=MODIFICATION
...
[8.5MiB/44.82s] Importing branch foobar-branch (dev-foobar-branch)
...
[179.7MiB/63.47s]   - Installing foobar/communication-lib (1.0.5): [179.7MiB/63.47s] Using HTTP basic authentication with username "user"
[179.7MiB/63.47s] Downloading https://***********/rest/api/1.0/projects/lib/repos/communication-lib/archive?at=42ed18843dfcf3313a33fc81a2e897b902f036b8&format=zip
[179.7MiB/63.47s] [179.7MiB/64.52s] Downloading (100%)[179.7MiB/64.52s]                   [179.7MiB/64.52s] /64.52s]
[179.7MiB/64.52s]  Extracting archive[179.7MiB/64.52s] Executing command (CWD): unzip -qq  '/test-composer/vendor/foobar/communication-lib/2473fbda90acc9b89780cdd0f4463cde' -d '/test-composer/vendor/composer/23d5bdb5'

Related #9100 #8388

@Seldaek Seldaek added this to the 2.x milestone Aug 22, 2020
@Seldaek
Copy link
Member

Seldaek commented Aug 22, 2020

Please rebase this on master branch, we will not add features to 1.x anymore.

@oleg-andreyev oleg-andreyev changed the base branch from 1.10 to master August 22, 2020 20:53

public static function supports(IOInterface $io, Config $config, $url, $deep = false)
{
return true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really not great.. as it means it'll be used as fallback for any URL not matching any other driver. I am not sure how to best fix it tbh, but it's be nice to have a way to identify this correctly, or maybe we force people to use the bitbucket-server type instead of magic vcs type if they want to use this.. That'd be ok too IMO, then supports is irrelevant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tell me what you think about the following solution:

public static function supports(IOInterface $io, Config $config, $url, $deep = false)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A solution might be to introduce a bitbuket-server-domains config setting, as already done to support Gitlab or Github Enterprise.

@oleg-andreyev
Copy link
Contributor Author

# Master
Command being timed: "/usr/local/bin/php /Users/oandreyev/Development/composer/bin/composer update --prefer-dist --ignore-platform-reqs --no-cache"
User time (seconds): 1.32
System time (seconds): 1.32
Percent of CPU this job got: 34%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.65
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 47588
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 206284
Voluntary context switches: 270
Involuntary context switches: 5168
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 139
Socket messages received: 854
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
# PR
Command being timed: "/usr/local/bin/php /Users/oandreyev/Development/composer/bin/composer update --prefer-dist --ignore-platform-reqs --no-cache"
User time (seconds): 2.05
System time (seconds): 0.72
Percent of CPU this job got: 6%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:41.17
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 50724
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 52660
Voluntary context switches: 410
Involuntary context switches: 6417
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 463
Socket messages received: 1601
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

image

Basically, CPU usage has dropped, but the timing has increased (but it's probably due to BB Server configuration and VPN)

@oleg-andreyev
Copy link
Contributor Author

# Master w/ cache
Command being timed: "/usr/local/bin/php /Users/oandreyev/Development/composer/bin/composer update --prefer-dist --ignore-platform-reqs"
User time (seconds): 0.70
System time (seconds): 0.78
Percent of CPU this job got: 26%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.57
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 37020
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 586
Minor (reclaiming a frame) page faults: 118948
Voluntary context switches: 277
Involuntary context switches: 4293
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 77
Socket messages received: 173
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
# PR w/ cache
Command being timed: "/usr/local/bin/php /Users/oandreyev/Development/composer/bin/composer update --prefer-dist --ignore-platform-reqs"
User time (seconds): 0.64
System time (seconds): 0.43
Percent of CPU this job got: 10%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 40512
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 48212
Voluntary context switches: 144
Involuntary context switches: 1852
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 117
Socket messages received: 284
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

image

Comment on lines +497 to +503
try {
$http = Factory::createHttpDownloader($io, $config);
$response = $http->get(sprintf('https://%s/rest/api/1.0/application-properties', $originUrl))->decodeJson();
} catch (\Exception $e) {
$io->writeError($e->getMessage());
return false;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not really acceptable to do an http request for every repository to chek if it's bitbucket server or not. This should work with a bitbucket-domains config value, similar to https://github.com/composer/composer/blob/master/src/Composer/Repository/Vcs/GitLabDriver.php#L527

protected $fallbackDriver;

/** @var string|null if set either git or hg */
protected $vcsType;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bitbucket Server currently only supports support for git repositories. This can be removed

protected $branches;
protected $infoCache = array();
protected $branchesUrl = '';
protected $tagsUrl = '';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$branchesUrl and $tagsUrl are unused and are not necessary. The Bitbucket Server API doesn't return those URLs so hard coding them as done in this PR should be ok.

protected $branchesUrl = '';
protected $tagsUrl = '';
protected $homeUrl = '';
protected $website = '';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$homeUrl and $website are never set. The Bitbucket Server API returns a self link which we can use and set as homepage in the Composer data.

}
$this->parseCloneUrls($repoData['links']['clone']);

$this->hasIssues = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is never set to true. I don't think Bitbucket Server has issues support. The API docs also do not list anything related. This can be removed.

return $this->branches;
}

protected function attemptCloneFallback()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is currently unused. You can have a look at other drivers to see how this is done. Basically you will want to extend the getContents method and handle cases where the credential doesn't have access to the API and then fallback to using git

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is then also the place where you can use the util class you created

protected function getRepoData()
{
$resource = sprintf(
'https://%s/rest/api/1.0/projects/%s/repos/%s',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All API URLs currently force HTTPS Bitbucket Server doesn't enforce HTTPS. I wonder whether we should handle this similar to the GitLabDriver where we detect the scheme from the repository URL?

@Seldaek Seldaek modified the milestones: 2.1, 2.2 May 24, 2021
@Seldaek Seldaek modified the milestones: 2.2, Nice To Have Nov 24, 2021
@beniamin
Copy link

I am interested in completing this PR. Is there anything I can help with?

@Seldaek
Copy link
Member

Seldaek commented Mar 21, 2022

@beniamin I'd say send your own PR based on this one, taking @glaubinix's feedback into account above, then we can hopefully move this forward?

@Rodrigostrong
Copy link


# Master w/ cache

Command being timed: "/usr/local/bin/php /Users/oandreyev/Development/composer/bin/composer update --prefer-dist --ignore-platform-reqs"

User time (seconds): 0.70

System time (seconds): 0.78

Percent of CPU this job got: 26%

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.57

Average shared text size (kbytes): 0

Average unshared data size (kbytes): 0

Average stack size (kbytes): 0

Average total size (kbytes): 0

Maximum resident set size (kbytes): 37020

Average resident set size (kbytes): 0

Major (requiring I/O) page faults: 586

Minor (reclaiming a frame) page faults: 118948

Voluntary context switches: 277

Involuntary context switches: 4293

Swaps: 0

File system inputs: 0

File system outputs: 0

Socket messages sent: 77

Socket messages received: 173

Signals delivered: 0

Page size (bytes): 4096

Exit status: 0


# PR w/ cache

Command being timed: "/usr/local/bin/php /Users/oandreyev/Development/composer/bin/composer update --prefer-dist --ignore-platform-reqs"

User time (seconds): 0.64

System time (seconds): 0.43

Percent of CPU this job got: 10%

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.05

Average shared text size (kbytes): 0

Average unshared data size (kbytes): 0

Average stack size (kbytes): 0

Average total size (kbytes): 0

Maximum resident set size (kbytes): 40512

Average resident set size (kbytes): 0

Major (requiring I/O) page faults: 0

Minor (reclaiming a frame) page faults: 48212

Voluntary context switches: 144

Involuntary context switches: 1852

Swaps: 0

File system inputs: 0

File system outputs: 0

Socket messages sent: 117

Socket messages received: 284

Signals delivered: 0

Page size (bytes): 4096

Exit status: 0

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants