Sorting complex git versions in bash for production update algorithm

Question

I am trying to fix a bug in an update bash script we have where we update a specific repository by grabbing and cherry picking every release tag in release order and commit->push it to a production repository. But the script stops at 1.9.0-1.1.0 (due to the ordering) and really messes things up when the current version is above 1.9.0-1.1.0.

But the git command which queries the remote tags order the tags in a bad way:

1.0.0-0.1.0
1.0.0-0.1.0
1.1.0-0.1.0
1.10.0-1.1.0
1.10.1-1.1.0
1.10.2-1.1.0
1.2.0-0.1.0
1.3.0-0.1.0
...
1.8.0-1.1.0
1.9.0-1.1.0

Note that I have cut a big chunk of tags between 1.3.0 and 1.8.0.

The version tags look like the following: 1.0.0-1.0.0, 1.0.1-1.0.0, 1.1.0-1.0.0, 1.2.0.1.0.0, 1.2.0-1.1.0 and so on, where the first part is the Base software, and the second part is the Engine modifications per client upon the base software, and thus we need to keep the versioning like so.

There is also another version where we have versions of 3 parts like: 1.2.0-1.1.0-1.0.1 This is also a way for us to track which Engine, which Engine modifications and then which Client modifications are present.

I have the following bash script:

echo "Add DEVELOPMENT remote repository."
git remote add development "https://gitlab.com/${DEVELOPMENT_REPO}"

echo "Fetch DEVELOPMENT repository master branch."
git fetch development master

LATEST_LOCAL_TAG=`git describe --abbrev=0 --tags`
echo "Latest local tag:"
echo $LATEST_LOCAL_TAG

REMOTE_TAGS=`git ls-remote --tags "https://gitlab.com/${DEVELOPMENT_REPO}" 2>&1`
echo "Remote tags:"

# Determines the tag can be concatenated to the final list or not.
CAN_PROCESS_TAG=false

# The pickable commit array.
CHERRY_PICKABLE_COMMITS=()

PROCESSED_TAGS=()

# Parse tags from remote and make cherry pickable tag list.
while IFS=' ' read -ra TAGS; do

    for tagData in "${TAGS[@]}"; do

        TAG_ORIGINAL=`sed -e 's|.*/\(.*\)$|\1|' <<< $tagData`
        TAG_VERSION=`sed -e 's|.*/\(.*\)$|\1|' <<< $tagData | tr -d '^{}'`

        if [[ "$TAG_VERSION" == "$LATEST_LOCAL_TAG" || $CAN_PROCESS_TAG == true ]]; then

            # Already processed such tag.
            if [[ " ${PROCESSED_TAGS[@]} " =~ " ${TAG_VERSION} " ]]; then
                continue
            fi

            if $CAN_PROCESS_TAG; then
                TAG_HASH=`sed 's|\(.*\)refs\/tags\/\(.*\)$|\1|' <<< $tagData | tr -d '\040\011\012\015'`

                TAG_VERSION_AS_MERGE="${TAG_VERSION}^{}"

                # If this is not a merge tag.
                if [[ $TAG_ORIGINAL != $TAG_VERSION_AS_MERGE ]]; then
                    grep "refs/tags/${TAG_VERSION_AS_MERGE}" <<< $REMOTE_TAGS;

                    # If there is a MERGE version of the tag, we skip this version. (The next version should be the merge one)
                    if [ "$?" == 0 ]; then
                        continue
                    fi
                fi

                # Determine if it is a merge.
                IS_MERGE=`git cat-file -p $TAG_HASH | grep -c "parent"`

                if [ $IS_MERGE == 1 ]; then
                    echo "Process commit: (${TAG_ORIGINAL}) ${TAG_HASH}"
                    git cherry-pick ${TAG_HASH} --no-commit --strategy-option=theirs
                else
                    echo "Process merge commit: (${TAG_ORIGINAL}) ${TAG_HASH}"
                    git cherry-pick ${TAG_HASH} --mainline 1 --no-commit --strategy-option=theirs
                fi;
            fi

            CAN_PROCESS_TAG=true

            # Processed tag, add it to our list.
            if [[ ! " ${PROCESSED_TAGS[@]} " =~ " ${TAG_VERSION} " ]]; then
                PROCESSED_TAGS=($TAG_VERSION)
            fi

        fi
    done

done <<< "$REMOTE_TAGS"

echo "Remove DEVELOPMENT remote."
git remote rm development

Sorting with REMOTE_TAGS=`git ls-remote --tags "https://gitlab.com/${DEVELOPMENT_REPO}" 2>&1 | sort --version-sort` will not do any good for me.

This is the RAW output into the REMOTE_TAGS variable:

e19eed8e0011d242c35f82cf27c5577256c34073 refs/tags/1.0.0-0.1.0 eccbd5365b8d0dba1cdea753297adab01a7bbd9f refs/tags/1.0.0-0.1.0^{} ae65488a0751891dea797bd4bbb891d011ec442a refs/tags/1.1.0-0.1.0 beb44666835c61c0925e0e7417df872cb48a8699 refs/tags/1.10.0-1.1.0 ae56780117e62b8abab2fff5b519ab8bf9087e50 refs/tags/1.10.1-1.1.0 b4ee505a141c33d9a1e935fe54313d014e309544 refs/tags/1.10.2-1.1.0 0c44aad7af050452dc97b3ab1d593482540038b5 refs/tags/1.11.0-1.1.0 c8eefc4c6016729b84ad19987874acb83f020c78 refs/tags/1.11.1-1.1.0 8af55cfd29be1435ca6d6c1196f402159f2b5477 refs/tags/1.11.2-1.1.0 e14d111632d9727be1b5d784bef5b492095b7292 refs/tags/1.11.3-1.1.0 3ed47e34b3f3508204778765af32444db2cd2a74 refs/tags/1.12.0-1.1.0 0549aa1037433cd4a5bf704889817ea1dc2ff7f4 refs/tags/1.12.1-1.1.0 3657945440a4d9ec0010bfb769a3686178749380 refs/tags/1.12.1-1.2.0 0be47905cf5554afadc2856d42327d1333be8e1f refs/tags/1.12.1-1.2.0^{} ba79fc39d458ba11191f15b2354a2d0c59d003fa refs/tags/1.2.0-0.1.0 31a2d8968095406bc7c6b8a75a8f01a637779c75 refs/tags/1.3.0-0.1.0 8a24a6ed8176d116da6085f72994965d2cce867b refs/tags/1.3.1-0.1.0 479359939c9fb075edbd8ee3926b610c1ea97d6c refs/tags/1.3.2-0.1.0 836c355395a435575a05d62ea6e7de6df15d3415 refs/tags/1.3.3-0.1.0 886aea61c22e5ec17b3d54010511d50cc71888e8 refs/tags/1.3.4-0.1.0 53aae6e285898b3d6a679389aaf2f2915639049f refs/tags/1.3.5-0.1.0 a686c687cf5a0b45b14578c43adf74c5c3858db3 refs/tags/1.3.6-0.1.0 6793b1908a383495780eb4c93164431cb03e9683 refs/tags/1.3.7-0.1.0 2896e6c517be095673bb2903e12e934ea81d7bdf refs/tags/1.3.8-0.1.0 14a786a83c1efa8112b346ba26f7c2b419a36ba0 refs/tags/1.3.8-0.1.1 291490820305aba05feae663ead75924cf2b91c8 refs/tags/1.3.8-0.1.1^{} da23ae28475ee64f98b0dea22d92518865a94b05 refs/tags/1.3.9-0.1.1 f1abd686b3ee5b274a3c282c95da4b8ef19607d9 refs/tags/1.4.0-0.1.1 3efe3fc6e92e403c0a36662e2fdc2863d92c6622 refs/tags/1.4.1-0.1.1 e5254b0ad13ad9ab4d2db5a3a1394b2949d1b452 refs/tags/1.4.2-0.1.1 f5c6c455f50dccc9367f0dd67610689acbce9ba6 refs/tags/1.4.2-0.1.1^{} e341268c4dd7d40d6e97429d0c1e64e92bd2448b refs/tags/1.4.2-0.1.2 9144fd16ab341d4eb88895f722164238f73b8c85 refs/tags/1.4.2-0.1.2^{} 862b511cb2889a45c04e3da3d3d713abeeb9631c refs/tags/1.4.2-0.2.0 3b4b460cf6a242f5bc766b6b53d079bd1b1a51e2 refs/tags/1.4.2-0.2.0^{} ecd720da97c943b6976666c712924cf6567777c4 refs/tags/1.5.0-0.2.0 2c7ba00d76093ec756567baec12e0854bbe8a94a refs/tags/1.6.0-0.2.0 6b6ccc4ab15697eeb5224b426cd7535107cd620a refs/tags/1.6.1-0.2.0 0c49dc835f06f4a93deb5175bfb744d496fff1c6 refs/tags/1.6.2-0.2.0 9ff0d0511191c1991264f889dff330d47829004f refs/tags/1.6.3-0.2.0 3e13ab8d970d5020611a9e714e4686dec539f838 refs/tags/1.6.3-1.0.0 26afdac1b6fa0fbb8935eb8ededb236028263ece refs/tags/1.6.3-1.0.0^{} 5339090c8d93b8dabb6ed6fca04ca0dc6472989d refs/tags/1.6.4-1.0.0 d8b71ddadd3103259936b9742c43396ade39a366 refs/tags/1.6.5-1.0.0 1537398f202404d1fb0f9f059be293dd32a780d4 refs/tags/1.7.0-1.0.0 3f3efef1356ad0cad3d79d1e99edf0ede3226aba refs/tags/1.7.0-1.1.0 f9c788895e6215bfc632ff0730ff9e4c4b126e47 refs/tags/1.7.0-1.1.0^{} 2a8360f7db63793d2f1e3cd0d42c38b67cb29ab7 refs/tags/1.7.1-1.1.0 791d808d1cce1416187b78f8969d533333dce779 refs/tags/1.7.2-1.1.0 e3a802a1ddd71b92c6b21d44bca4001717d597f6 refs/tags/1.8.0-1.1.0 c89018dfa70081121ebb0f84544a3c91a0604c43 refs/tags/1.9.0-1.1.0

Need to find a way to properly sort the above versions to get the following result:

1.0.0-0.1.0
1.1.0-0.1.0
1.2.0-0.1.0
1.3.0-0.1.0
...
1.8.0-1.1.0
1.9.0-1.1.0
1.10.0-1.1.0
1.10.1-1.1.0
1.10.2-1.1.0

Any help is welcome.

Edit: Doing

REMOTE_TAGS=`git ls-remote --tags "https://gitlab.com/${DEVELOPMENT_REPO}" 2>&1 | sed 's|.*/\(.*\)$|\1|' | grep -v '\^' | tr "-" . | sort -t. -k1,1nr -k2,2nr -k3,3nr -k4,4nr -k5,5nr -k6,6nr -k7,7nr`

Will sort my versions properly, but will remove the hashes which is required for my cherry pick. Also replaces - which needs to be re-added afterwards somehow. Any idea how to re-map it back after this sort? Maybe iterating through this result set and grepping from the original?

Answer 1

Git can sort refs with version:refname (shorter: v:refname) like this:

> git ls-remote --tags --sort=v:refname $URL
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.0.0-0.1.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.1.0-0.1.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.1.0-0.9.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.1.0-0.10.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.2.0-0.1.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.3.0-0.1.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.8.0-1.1.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.9.0-1.1.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.10.0-1.1.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.10.1-1.1.0
da267f78ec24d210b1b95a266f55266767be2290    refs/tags/1.10.2-1.1.0