Process only changed files

Question

What:
With jenkins I want to process periodically only changed files from SVN and commit the output of the processing back to SVN.

Why:
We are committing binary files into SVN (we are working with Oracle Forms and are committing fmb-Files). I created a script which exports the fmb's to xml (with the original Fmb2XML tool from Oracle) and than I convert the XML to plain source which we also want to commit. This allows us greping, viewing the changes, ....

Problem:
At the moment I am only able to checkout everything, convert the whole directory and committing the whole directory back to SVN. But since all plain text files are newly generated they appear changed in SVN. I want to commit only the changed ones.

Can anyone help me with this?

Answer 1

I installed the Groovy plugin, configured the Groovy language and created a script which I execute as "system Groovy script". The scripts looks like:

import java.lang.ProcessBuilder.Redirect

import hudson.model.*
import hudson.util.*
import hudson.scm.*
import hudson.scm.SubversionChangeLogSet.LogEntry

// uncomment one of the following def build = ... lines

// work with current build
def build = Thread.currentThread()?.executable

// for testing, use last build or specific build number
//def item = hudson.model.Hudson.instance.getItem("Update_SRC_Branch") 
//def build = item.getLastBuild()   
//def build = item.getBuildByNumber(35)   

// get ChangesSets with all changed items
def changeSet= build.getChangeSet()
List<LogEntry> items = changeSet.getItems()

def affectedFiles = items.collect { it.paths }

// get filtered file names (only fmb) without path
def fileNames = affectedFiles.flatten().findResults {
    if (it.path.substring(it.path.lastIndexOf(".") + 1) != "fmb") return null
    it.path.substring(it.path.lastIndexOf("/") + 1)
}.sort().unique()

// setup log files
def stdOutFile = "${build.rootDir}\\stdout.txt"
def stdErrFile = "${build.rootDir}\\stderr.txt"

// now execute the external transforming
fileNames.each {
    def params = [...]
    def processBuilder = new ProcessBuilder(params)

    // redirect stdout and stderr to log files
    processBuilder.redirectOutput(new File(stdOutFile))
    processBuilder.redirectError(new File(stdErrFile))

    def process = processBuilder.start()
    process.waitFor()

    // print log files
    println new File(stdOutFile).readLines()
    System.err.println new File(stdErrFile).readLines()
}

Afterwards I use command line with "svn commit" to commit the updated files.

Answer 2

Preliminary note: getting files from repo in SVN-jargon is "checkout", saving to repo - "commit". Don't mix CVS and SVN terms, it can lead to misinterpretation

In order to get list of changed files in revision (or revset) you can use

• Easy way - svn log with options -q -v. For single revision you also add -c REVNO, for revision range: -r REVSTART:REVEND. Probably additional --xml will produce more suitable output, than plain-text

You must to post-process output of log in order to get pure list, because: log contain some useless for you data, in case of log for range you can have the same file included in more than one revision

z:\>svn log -q -v -r 1190 https://subversion.assembla.com/svn/customlocations-greylink/
------------------------------------------------------------------------
r1190 | lazybadger | 2012-09-20 13:19:45 +0600 (Чт, 20 сен 2012)
Changed paths:
   M /trunk/Abrikos.ini
   M /trunk/ER-Telecom.ini
   M /trunk/GorNet.ini
   M /trunk/KrosLine.ini
   M /trunk/Rostelecom.ini
   M /trunk/Vladlink.ini
------------------------------------------------------------------------

example of single revision: you have to log | grep trunk | sort -u, add repo-base to filenames

• Harder way: with additional SCM (namely - Mercurial) and hgsubversion you'll get slightly more (maybe) log with hg log --template "{files}\n" - only slightly because you'll get only filelist, but filesets in different revisions are newline-separated, filenames inside revision are space-separated