Jenkins export and import jobs 迁移导出导入任务实践小结

前言

我遇到的Jenkins迁移项目并非可以通过简单的物理文件复制就可以轻松解决,需要考虑上千个不同项目的jobs分离,Jenkins 1.x和2.x大版本兼容性,Jenkins Plugins插件,Jenkins Credentials凭证,Jenkins Restrict节点约束,按view分类不同项目的jobs等各种因素。这次对Jenkins迁移做了大量的研究和实践,希望总结出来的方法能对各位有帮助。

Jenkins export and import jobs 迁移导出导入任务实践小结

更新历史

2020年01月06日 - 初稿

阅读原文 - https://wsgzao.github.io/post...

扩展阅读

Jenkins Plugins Index


export and import jobs in Jenkins

Is it possible to exchange jobs between 2 different Jenkins'? I'm searching for a way to export/import jobs.

Solutions/Answers:

Solution 1:

Jenkins has a rather good wiki, albeit hard to read when you’re new to CI software…

They offer a simple solution for moving jobs between servers

The trick probably was the need to reload config from the Jenkins Configuration Page.

Solution 2:

Probably use jenkins command line is another option, see https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+CLI

  • create-job: Creates a new job by reading stdin as a configuration XML file.
  • get-job: Dumps the job definition XML to stdout

So you can do

java -jar jenkins-cli.jar -s http://server get-job myjob > myjob.xml
java -jar jenkins-cli.jar -s http://server create-job newmyjob < myjob.xml

It works fine for me and I am used to store in inside my version control system

Solution 3:

A one-liner:

$ curl -s http://OLD_JENKINS/job/JOBNAME/config.xml | curl -X POST 'http://NEW_JENKINS/createItem?name=JOBNAME' --header "Content-Type: application/xml" -d @-

With authentication:

$ curl -s http:///<USER>:<API_TOKEN>@OLD_JENKINS/job/JOBNAME/config.xml | curl -X POST 'http:///<USER>:<API_TOKEN>@NEW_JENKINS/createItem?name=JOBNAME' --header "Content-Type: application/xml" -d @-

With Crumb, if CSRF is active (see details here):

Get crumb with:

$ CRUMB_OLD=$(curl -s 'http://<USER>:<API_TOKEN>@OLD_JENKINS/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
$ CRUMB_NEW=$(curl -s 'http://<USER>:<API_TOKEN>@NEW_JENKINS/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')

Apply crumb with -H CRUMB:

$ curl -s -H $CRUMB_OLD http:///<USER>:<API_TOKEN>@OLD_JENKINS/job/JOBNAME/config.xml | curl -X POST -H $CRUMB_NEW 'http:///<USER>:<API_TOKEN>@NEW_JENKINS/createItem?name=JOBNAME' --header "Content-Type: application/xml" -d @-

Solution 4:

There’s a plugin called Job Import Plugin that may be what you are looking for. I have used it. It does have issues with importing projects from a server that doesn’t allow anonymous access.

For Completeness:
If you have command line access to both, you can do the procedure already mentioned by Khez for Moving, Copying and Renaming Jenkins Jobs.

Solution 5:

Go to your Jenkins server’s front page, click on REST API at the bottom of the page:

Create Job

To create a new job, post config.xml to this URL with query parameter name=JOBNAME. You need to send a Content-Type: application/xml header. You’ll get 200 status code if the creation is successful, or 4xx/5xx code if it fails. config.xml is the format Jenkins uses to store the project in the file system, so you can see examples of them in the Jenkins home directory, or by retrieving the XML configuration of existing jobs from /job/JOBNAME/config.xml.

Solution 6:

In my Jenkins instance (version 1.548) the configuration file is at:

/var/lib/jenkins/jobs/-the-project-name-/config.xml

Owned by jenkins user and jenkins group with 644 permissions. Copying the file to and from here should work. I haven’t tried changing it directly but have backed-up the config from this spot in case the project needs to be setup again.

Solution 7:

Job Import plugin is the easy way here to import jobs from another Jenkins instance. Just need to provide the URL of the source Jenkins instance. The Remote Jenkins URL can take any of the following types of URLs:

  • http://$JENKINS – get all jobs on remote instance
  • http://$JENKINS/job/$JOBNAME – get a single job
  • http://$JENKINS/view/$VIEWNAME – get all jobs in a particular view

Solution 8:

Thanks to Larry Cai’s answer I managed to create a script to backup all my Jenkins jobs. I created a job that runs this every week. In case someone finds it useful, here it is:

#!/bin/bash
#IFS for jobs with spaces.
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for i in $(java -jar /run/jenkins/war/WEB-INF/jenkins-cli.jar -s http://server:8080/ list-jobs);
do
  java -jar /run/jenkins/war/WEB-INF/jenkins-cli.jar -s http://server:8080/ get-job ${i} > ${i}.xml;
done
IFS=$SAVEIFS
mkdir deploy
tar cvfj "jenkins-jobs.tar.bz2" ./*.xml

Solution 9:

Jenkins export jobs to a directory

 #! /bin/bash
    SAVEIFS=$IFS
    IFS=$(echo -en "\n\b")
    declare -i j=0
    for i in $(java -jar jenkins-cli.jar -s http://server:8080/jenkins list-jobs  --username **** --password ***);
    do
    let "j++";
    echo $j;
    if [ $j -gt 283 ] // If you have more jobs do it in chunks as it will terminate in the middle of the process. So Resume your job from where it ends.
     then
    java -jar jenkins-cli.jar -s http://lxvbmcbma:8080/jenkins get-job --username **** --password **** ${i} > ${i}.xml;
    echo "done";
    fi
    done

Import jobs

for f in *.xml;
do
echo "Processing ${f%.*} file.."; //truncate the .xml extention and load the xml file for job creation
java -jar jenkins-cli.jar -s http://server:8080/jenkins create-job ${f%.*}  < $f
done

Solution 10:

Simple php script worked for me.

Export:

// add all job codes in the array
$jobs = array("job1", "job2", "job3");

foreach ($jobs as $value)
{
    fwrite(STDOUT, $value. " \n") or die("Unable to open file!");
    $path = "http://server1:8080/jenkins/job/".$value."/config.xml";
    $myfile = fopen($value.".xml", "w");
    fwrite($myfile, file_get_contents($path));
    fclose($myfile);
}

Import:

<?php

// add all job codes in the array
$jobs = array("job1", "job2", "job3");

foreach ($arr as $value)
{
    fwrite(STDOUT, $value. " \n") or die("Unable to open file!");
    $cmd = "java -jar jenkins-cli.jar -s http://server2:8080/jenkins/ create-job ".$value." < ".$value.".xml";
    echo exec($cmd);
}

Solution 11:

This does not work for existing jobs, however there is Jenkins job builder.

This allows one to keep job definitions in yaml files and in a git repo which is very portable.

Solution 12:

For those of us in the Windows world who may or may not have Bash available, here’s my PowerShell port of Katu and Larry Cai‘s approach. Hope it helps someone.

##### Config vars #####
$serverUri = 'http://localhost:8080/' # URI of your Jenkins server
$jenkinsCli = 'C:\Program Files (x86)\Jenkins\war\WEB-INF\jenkins-cli.jar' # Path to jenkins-cli.jar on your machine
$destFolder = 'C:\Jenkins Backup\' # Output folder (will be created if it doesn't exist)
$destFile = 'jenkins-jobs.zip' # Output filename (will be overwritten if it exists)
########################

$work = Join-Path ([System.IO.Path]::GetTempPath()) ([System.IO.Path]::GetRandomFileName())
New-Item -ItemType Directory -Force -Path $work | Out-Null # Suppress output noise
echo "Created a temp working folder: $work"

$jobs = (java -jar $jenkinsCli -s $serverUri list-jobs)
echo "Found $($jobs.Length) existing jobs: [$jobs]"

foreach ($j in $jobs)
{
    $outfile = Join-Path $work "$j.xml"
    java -jar $jenkinsCli -s $serverUri get-job $j | Out-File $outfile
}
echo "Saved $($jobs.Length) jobs to temp XML files"

New-Item -ItemType Directory -Force -Path $destFolder | Out-Null # Suppress output noise
echo "Found (or created) $destFolder folder"

$destPath = Join-Path $destFolder $destFile
Get-ChildItem $work -Filter *.xml |
    Write-Zip -Level 9 -OutputPath $destPath -FlattenPaths |
    Out-Null # Suppress output noise
echo "Copied $($jobs.Length) jobs to $destPath"

Remove-Item $work -Recurse -Force
echo "Removed temp working folder"

Solution 13:

It is very easy just download plugin name

Job Import Plugin

Enter the URL of your Remote Jenkins server and it will import the jobs automatically

Solution 14:

The most easy way, with direct access to the machine is to copy the job folder from first jenkins to another one (you can exclude workspaces – workspace folder), because the whole job configuration is stored in the xml file on the disk.

Then in the new jenkins just reload configuration in the global settings (admin access is required) should be enough, if not, then you will need to restart Jenkins tool.

Another way can be to use plugins mentioned above this post.

edit:
– in case you can probably also exclude modules folders

Jenkins迁移方案小结

上面列举有14种方法居多,但大多数是基于方案4进行扩展

上面列举的方案基本已经非常全了,感觉是不是有点眼花?其实归纳总结主要有以下4种

  1. 官方的Moving/copying/renaming jobs,即所谓的物理文件迁移,如果情况复杂此方案不推荐
  2. 基于Jenkins CLI,该方案需要依赖jenkins-cli.jar包括java使用范围具有一定局限性,不推荐
  3. 基于类似Job Import Plugin插件形式,该类方案普适性较差,不推荐
  4. 基于Jenkins REST API,核心是获取<jenkinshost>/job/<jobname>/config.xml,效果最佳

REST API

Many objects of Jenkins provide the remote access API. They are available at /.../api/ where "..." portion is the object for which you'd like to access.

XML API

Access data exposed in HTML as XML for machine consumption. Schema is also available.
You can also specify optional XPath to control the fragment you'd like to obtain (but see below). For example, ../api/xml?xpath=//[0].

For XPath that matches multiple nodes, you need to also specify the "wrapper" query parameter to specify the name of the root XML element to be create so that the resulting XML becomes well-formed.

Similarly exclude query parameter can be used to exclude nodes that match the given XPath from the result. This is useful for trimming down the amount of data you fetch (but again see below). This query parameter can be specified multiple times.

XPath filtering is powerful, and you can have it only return a very small data, but note that the server still has to build a full DOM of the raw data, which could cause a large memory spike. To avoid overloading the server, consider using the tree parameter, or use the xpath parameter in conjunction with the tree parameter. When used together, the result of the tree parameter filtering is built into DOM, then the XPath is applied to compute the final return value. In this way, you can often substantially reduce the size of DOM built in memory.

JSON API

Access the same data as JSON for JavaScript-based access. tree may be used.

Python API

Access the same data as Python for Python clients. This can be parsed into Python object as eval(urllib.urlopen("...").read()) and the resulting object tree is identical to that of JSON. However, when you do this, beware of the security implication. If you are connecting to a non-trusted Jenkins, the server can send you malicious Python programs.

In Python 2.6 or later you can safely parse this output using ast.literal_eval(urllib.urlopen("...").read())

For more information about remote API in Jenkins, see the documentation.

https://jenkins.io/redirect/r...

使用cURL和Jenkins REST API

使用Jenkins REST API建议关闭CSRF防护
# To retrieve the job config.xml
curl -X GET '<jenkinshost>/job/<jobname>/config.xml' -u username:API_TOKEN -o <jobname>.xml

# to use this config to create a new job
curl -s -XPOST '<jenkinshost>/createItem?name=<jobname>' -u username:API_TOKEN --data-binary @<jobname>.xml -H "Content-Type:text/xml"

# get all jenkins jobs
curl -X GET '<jenkinshost>/api/json?pretty=true' -u username:API_TOKEN -o jobs.json

# get jenkins view
curl -X GET '<jenkinshost>/view/<viewname>/api/json' -u username:API_TOKEN -o view.json

Obviously, replace:

  • username API_TOKEN with your username and password/API_Token
  • jenkinshost with your Jenkins URL
  • jobname with the name of the job that you created via the UI

JSON to CSV Converter

mkdir config_xml
vim jobs.txt
vim jenkins_jobs_migration.sh

#/bin/bash
source_jenkins_host="xxx"
source_jenkins_username="xxx"
source_jenkins_password="xxx"

target_jenkins_host="xxx"
target_jenkins_username="xxx"
target_jenkins_password="xxx"

pull() {
    for line in `cat jobs.txt`
    do
        echo ${line}
        curl -X GET ${source_jenkins_host}job/${line}/config.xml -u ${source_jenkins_username}:${source_jenkins_password} -o config_xml/${line}.xml
    done
}

push() {
    for line in `cat jobs.txt`
    do
        echo $line
        curl -s -XPOST ${target_jenkins_host}createItem?name=${line} -u ${target_jenkins_username}:${target_jenkins_password} --data-binary @config_xml/${line}.xml -H "Content-Type:text/xml"
    done

}

case "$1" in
    push)
        push
        ;;
    pull)
        pull
        ;;
    *)
        echo "Usage: $NAME {pull|push}"
        exit 1
        ;;
esac

使用python编写api_jenkins.py核心逻辑

暂不方便公开
  1. 支持读取context.json配置文件使用pipeline.j2模板自动生成不同类型的pipeline.xml导入文件
  2. 支持按view或者job export导出job config.xml配置信息
  3. 支持按view或者job import导入job config.xml配置信息

Jenkins Backup

archive jenkins setting and plugins

大家应该都知道Jenkins备份插件目前主要就2种选择:

严格意义上来说应该只能选择Periodic Backup,但是如果是需要定期备份自然离不开编写Bash脚本

jenkins-backup-script

参考文章

Export/import jobs in Jenkins

Moving/copying/renaming jobs

Jenkins CLI

How to create a job using the REST API and cURL?

Jenkins Remote access API

Image placeholder
xiaofengye
未设置
  30人点赞

没有讨论,发表一下自己的看法吧

推荐文章
Docker Compose 安装 Jenkins

前言上回我们已经安装好了Docker,这篇我们利用Compose,通过配置docker-compose.yml安装jenkins。之所以使用Compose,是因为后续如果有更多容器,能够更为方便直观的

分布式事务实战

引言:微服务倡导将复杂的单体应用拆分为若干个功能简单、松耦合的服务,这样可以降低开发难度、增强扩展性、便于敏捷开发,从而被越来越多的开发者和公司推崇运用。但系统微服务化后,一个看似简单的功能,内部可能

maatwebsite/Excel 3.1 使用教程 (导入篇)

maatwebsite/excel使用教程(导入篇) 官方文档 https://docs.laravel-excel.com/3.1/getting... GIT地址 https://github.c

Jenkins CLI 命令行 v0.0.23

本文首发于:Jenkins中文社区作者:ZhaoXiaojieJenkinsCLI在参加2019年谁是最受欢迎的中国开源软件投票,如果您已经是JenkinsCLI的用户,请点击下面的链接帮忙投上一票。

在Jenkins中发布react 静态项目常用shell脚本

功效:服务器git永远同步,远程,本地有更改文件或文件夹会被重置掉,保持和远程仓库一致 rsync同步文件 永久链接:https://shudong.wang/10705.html获取git远程仓库

使用Jenkins一键打包部署SpringBoot应用,就是这么6!

SpringBoot实战电商项目mall(25k+star)地址:https://github.com/macrozheng/mall 摘要任何简单操作的背后,都有一套相当复杂的机制。本文将以Spri

MVVM原理(Object.defineProperty和订阅者模式)

想着去了解vue的mvvm数据驱动是怎么实现的,百度中看了这篇文章,demo很好。其他文章只是讲到defineProperty的set,get。彻底理解Vue中的Watcher、Observer、De

集成 think-ORM 的 symfony bundle thinkorm-bundle

thinkorm-bundleSymfonyThinkOrmBundle关于thinkorm-bundle允许在你symfony使用thinkorm.所安装$composerrequireccwwwo

elastic-job-lite 使用的一点心得和坑

elastic-job-lite开篇词 elastic-job-lite在项目中使用也有两个多月的时间了,从一开始搜索网上教程,参考别人使用方法,到后面阅读源码,理解其架构,实现。也写了几篇关于ej

两年Flink迁移之路:从standalone到on yarn,处理能力提升五倍

一、背景与痛点在2017年上半年以前,TalkingData的AppAnalytics和GameAnalytics两个产品,流式框架使用的是自研的td-etl-framework。该框架降低了开发流式

从Oracle到PostgreSQL,某保险公司迁移实践

摘要:去O一直是金融保险行业永恒的话题,但去O的难度之大也只有真正经历过的人才知其中的艰辛。此次笔者结合实际去O工作,对去O过程中碰到的DBLINK、SEQUENCE最大值、空串、SQL语句中的别名等

从 Oracle 到 PostgreSQL ,某保险公司迁移实践

作者 |章晨曦编辑 | 老鱼摘要:去O一直是金融保险行业永恒的话题,但去O的难度之大也只有真正经历过的人才知其中的艰辛。此次笔者结合实际去O工作,对去O过程中碰到的DBLINK、SEQUENCE最大值

小米Kylin平滑迁移HBase实践

根据美团等其他公司在Kylin社区的公开分享资料,跨HBase集群升级方案需要在新集群重新构建历史的Cube,或者有一段时间的服务停止。小米在Kylin生产环境的跨HBase集群迁移中实现了无中断的平

宜信微服务任务调度平台建设实践|分享实录

导读:如今,无论是互联网应用还是企业级应用,都充斥着大量的批处理任务,常常需要一些任务调度系统帮助我们解决问题。随着微服务化架构的逐步演进,单体架构逐渐演变为分布式、微服务架构。内容来源:宜信技术学院

宜信开源|微服务任务调度平台SIA-TASK入手实践

引言最近宜信开源微服务任务调度平台SIA-TASK,SIA-TASK属于分布式的任务调度平台,使用起来简单方便,非常容易入手,部署搭建好SIA-TASK任务调度平台之后,编写TASK后配置JOB进行调

Express 官网文档翻译-2.4-基础路由

基本路由 路由指确定应用程序如何响应客户端对特定终结点的请求,它是一个URI(或路径)和一个特定的HTTP请求方法(GET、POST等)。 每个路由都可以有一个或多个处理程序函数,这些函数在路由匹配

Express 官网文档翻译-3.2-开发中间件

为Express应用开发中间件概述中间件函数本质上是一些可以在应用的请求-响应周期内,访问请求对象 (req),响应对象 (res),和next方法的函数。next方法是Express路由中的一个方法

Express 官网文档翻译-3.3-使用中间件

使用中间件Express框架是一个由路由和中间件构成的web开发框架,它内置的功能很少:Express应用程序本质上是一系列中间件函数调用。中间件 函数本质上是一些可以在应用的请求-响应周期内,访问 

正则表达式 Regular Expression

历史正则表达式出现于理论计算机科学的自动控制理论和形式化语言理论中。在这些领域中有对计算(自动控制)的模型和对形式化语言描述与分类的研究。1它可以转化成形式化语言或者确定型自动机。它们是语义上等价的,

JetBrains开发工具正版授权领取

大牛新录制的全套js课程疫情免费赠送领取地址如下: 课程一:JavaScript基础全套实战教程 课程二:JavaScript的WebAPI之BOM操作 课程三:JavaScript的WebAPI之D

03.5. 小结

这一章我们介绍了HTTP协议,DNS解析的过程,如何用go实现一个简陋的webserver。并深入到net/http包的源码中为大家揭开实现此server的秘密。 希望通过这一章的学习,你能够对Go开

GoWeb教程_ 04.6. 小结

这一章里面我们学习了Go如何处理表单信息,我们通过用户登录、上传文件的例子展示了Go处理form表单信息及上传文件的手段。但是在处理表单过程中我们需要验证用户输入的信息,考虑到网站安全的重要性,数据过

GoWeb教程_ 05.7. 小结

这一章我们讲解了Go如何设计database/sql接口,然后介绍了各种第三方关系型数据库驱动的使用。接着介绍了beedb,一种基于关系型数据库的ORM库,如何对数据库进行简单的操作。最后介绍了NOS

GoWeb教程_06.5. 小结

这章我们学习了什么是session,什么是cookie,以及他们两者之间的关系。但是目前Go官方标准包里面不支持session,所以我们设计了一个session管理器,实现了session从创建到销毁

GoWeb教程_07.7. 小结

这一章给大家介绍了一些文本处理的工具,包括XML、JSON、正则和模板技术,XML和JSON是数据交互的工具,通过XML和JSON你可以表达各种含义,通过正则你可以处理文本(搜索、替换、截取),通过模