Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Jenkins Git Parameter plugin cannot fetch tags

Tags:

git

ssh

jenkins

  • Jenkins version: 1.593
  • Git Parameter Plug-In: 0.4.0
  • GIT client plugin: 1.16.1

I use a private git repository, accessible over ssh.

My build is parameterized. The git parameter is TAG_TO_BUILD, the branch to build is refs/tags/${TAG_TO_BUILD}

After the initial build of the workspace, when I start a build with parameters, I get noWorkspaceError.

In the Jenkins log I have

Feb 16, 2015 2:17:34 PM WARNING org.eclipse.jetty.util.log.JavaUtilLog warn
Error while serving http://foo/view/bar/job/baz/descriptorByName/net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition/fillValueItems
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
    at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:46)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    at org.jenkinsci.plugins.modernstatus.ModernStatusFilter.doFilter(ModernStatusFilter.java:52)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress git@foo:foobar/foobaz.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
fatal: The remote end hung up unexpectedly

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1591)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1379)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324)
    at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.generateContents(GitParameterDefinition.java:314)
    at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition$DescriptorImpl.doFillValueItems(GitParameterDefinition.java:536)
    ... 85 more

I am using ssh username with private key credentials, the private key is entered directly. I can be certain that the key is correct, because the job can be built without parameters.

So it seems that the fetching of tags borks on this command but I don't know why: git -c core.askpass=true fetch --tags --progress git@foo:foobar/foobaz.git +refs/heads/*:refs/remotes/origin/* If I execute this command on my local machine, with a ~/.ssh/config file that points to my own private key, then that command does not error out.

Any hints on troubleshooting? I cannot log in on the Jenkins server as the jenkins user because it is a hosted server, so that's off the table.

like image 405
Amedee Van Gasse Avatar asked Mar 18 '23 00:03

Amedee Van Gasse


2 Answers

I could not get this to work. It's confirmed to be a SSH issue (see comment on question), and I don't have access to the jenkins user so I can't create a ~/.ssh/config for it.

I solved it in another way:

  • I installed the Extensible Choice Parameter plugin
  • Parameter Name: TAG_TO_BUILD
  • Choice provider: System Groovy Choice Parameter
  • Groovy System Script

.

def feedUrl = 'URL_TO_PHP_SCRIPT'
def gettags = new URL(feedUrl).getText()
def tags = []
def t1 = []
gettags.eachLine {tags.add(it)}
for(i in tags)
    t1.add(i.split()[1].replaceAll('\\^\\{\\}','').replaceAll('refs/tags/', ''))
t1 = t1.unique().sort().reverse()
return t1
  • PHP script on a server that is accessible to Jenkins (happens to be the same server but with nginx running as a different user as jenkins, and I do have shell access so I can create a ~/.ssh/config):

.

<?php
exec('git ls-remote -t GIT_URL', $output);
print(preg_replace('/[a-z0-9]*\trefs\/heads\//','',implode("\n", $output)));
?>
  • ~/.ssh/config on the user account that runs the webserver:

.

Host    GIT_SERVER
    IdentityFile    PATH_TO_SSH_PRIVATE_KEY
    IdentitiesOnly  yes
  • The corresponding SSH public key was added to the Git server.

The disadvantage of this setup is that it is very cludgy. Just using the Git Parameter plugin would have been a lot more elegant, but that is unfortunately broken for me.

The main advantage for me is that I could test every individual part of it:

  1. The git ls-remote command in a shell: outputs SHAs and tag references.
  2. php getgittags.php outputs the list of all tags to the shell.
  3. Browse to getgittags.php and verify I got the same tag list.
  4. The Extensible Choice Parameter has a "Run the Script Now" button, allowing you to test before you save your changes.
  5. Finally, start a parameterized build.

I took ideas from these StackOverflow answers:

  • https://stackoverflow.com/a/15243393/766786
  • https://stackoverflow.com/a/18112484/766786
like image 132
Amedee Van Gasse Avatar answered Mar 20 '23 07:03

Amedee Van Gasse


/**
 * Purpose: 
 * Listing git tags with the Jenkins Git Parameters plugin for a private repo just doesn't work while you are also using a jenkins pipeline file from scm and the pipeline file is not in the repo that you want to list the tags for (because the pipeline file is shared across multiple applications)
 * This script will solve all your tag listing problems
 * 
 * Setup steps:
 * 1. Create a credentials id for a https username and password for the git repo you want the tags for
 * 2. Past this code into the Groovy script for your active choices parameter
 * 3. Update the credentialsId and httpsRepo variables below
 * 4. Authorize any scripts that are being denied in the https://yourjenkinsurl:8080/scriptApproval/
 * 
 * This will NOT work with ssh. Don't even think about it!
 * However, if you do figure out how to get his to work with ssh. Please do ping me
 */
import jenkins.model.*
import java.net.URLEncoder;

// The credentials id with the username and password that will authenticate with the https git repo
String credentialsId = 'gitlab'

// The https repo url without 'https://'
String httpsRepo = 'gitlab.com/cryptexlabs/private/redis.git'

// A grep filter for the tags. If you don't want a filter at all then just delete the ` | grep ${filter}` statement below
String filter = "v"

def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
  com.cloudbees.plugins.credentials.common.StandardUsernameCredentials.class, Jenkins.instance, null, null ).find{
    it.id == credentialsId}

String encodedUser = URLEncoder.encode(creds.username, "UTF-8")
String encodedPassword = URLEncoder.encode(creds.password, "UTF-8")

def proc = "git ls-remote --tags https://${encodedUser}:${encodedPassword}@${httpsRepo}".execute() | 'cut -d/ -f3-'.execute() | 'sort -t. -nk1,2 -k3'.execute() | "grep ${filter}".execute() | 'grep -v {'.execute()
proc.waitFor()
def output = proc.in.text
return output.tokenize()
like image 42
Josh Woodcock Avatar answered Mar 20 '23 05:03

Josh Woodcock