Очистка места в gitlab-registry

Thank you for reading this post, don't forget to subscribe!

Не так давно обнаружились проблемы с очисткой места, которое занимают собранные docker-образы в настроенном нами docker-registry

Выяснилось, что в веб-интерфейсе Gitlab на вкладке Registry при нажатии на кнопку «удалить» на самом деле происходит удаление только тэга образа, а сами данные никуда не деваются и продолжают занимать место на жестком диске.

Аналогичная проблема наблюдается если docker-образ создается с одним и тем же тэгом (например, latest) — а это обычное дело для процесса CI. В этом случае, при сборке нового образа старый не удаляется (как ожидается), у него просто пропадает тэг (latest).

Например, у образа master/lebed/test:latest на самом деле на жестком диске хранится 2 версии:

ls -la /srv/gitlab/shared/registry/docker/registry/v2/repositories/master/lebed/<span class="hljs-built_in">test</span>/_manifests/tags/latest/index/sha256
total 16
drwxr-xr-x 4 root root 4096 Jun 15 12:20 .
drwxr-xr-x 3 root root 4096 Jun 15 09:25 ..
drwxr-xr-x 2 root root 4096 Jun 15 12:20 694a3871d67d64d5c59d3f6943be3cfc0d11aae2082261735e87b1c3c58bd66b
drwxr-xr-x 2 root root 4096 Jun 15 09:25 e1f5741ced3f0280bd372baa6ca293d1033dfefc7db021<span class="hljs-built_in">cd</span>81ca1527d2f1c08e

ls -la /srv/gitlab/shared/registry/docker/registry/v2/repositories/master/lebed/<span class="hljs-built_in">test</span>/_manifests/tags/latest/index/sha256

total 16

drwxr-xr-x 4 root root 4096 Jun 15 12:20 .

drwxr-xr-x 3 root root 4096 Jun 15 09:25 ..

drwxr-xr-x 2 root root 4096 Jun 15 12:20 694a3871d67d64d5c59d3f6943be3cfc0d11aae2082261735e87b1c3c58bd66b

drwxr-xr-x 2 root root 4096 Jun 15 09:25 e1f5741ced3f0280bd372baa6ca293d1033dfefc7db021<span class="hljs-built_in">cd</span>81ca1527d2f1c08e

При частых сборках docker-образов отведенное место в docker-registry может закончиться — при этом невозможно будет пушить в него новые образы и весь процесс CI перестанет работать.

Для очистки места в приватном docker-registry необходимо выполнить следующие действия:

удалить старые версии тегов docker-образа;
удалить старые версии ревизий docker-образа;
запустить процесс «уборки мусора» в контейнере с docker-registry.

Рассмотрим подробнее пример выполнения данных действий. Находясь на docker-хосте, смотрим содержимое каталога с интересующим нас docker-образом и его тэгом в каталоге

/srv/gitlab/shared/registry/docker/registry/v2/repositories/{имя_докер_образа}/_manifests/tags/{тег_образа}/index/sha256

В нем должны находиться один или несколько каталогов вида

e1f5741ced3f0280bd372baa6ca293d1033dfefc7db021cd81ca1527d2f1c08e.

Если каталог один, то никаких действий производить не следует, если же таких директорий несколько, то нужно удалить все старые версии, оставив только одну, самую последнюю по дате создания.

Смотрим содержимое каталога с ревизиями интересующего нас docker-образа

/srv/gitlab/shared/registry/docker/registry/v2/repositories/{имя_докер_образа}/_manifests/revisions/sha256

В нем должны также быть вложены директории с именами вида

e1f5741ced3f0280bd372baa6ca293d1033dfefc7db021cd81ca1527d2f1c08e — точно такие же, как и в предыдущем шаге. Удаляем более старые (ориентируемся по дате создания), оставляя только один каталог.

После удаления вышеуказанных файлов запускаем процесс «уборки мусора» командой:

docker exec -it {имя_контейнера_c_registry} bin/registry garbage-collect {путь_к_конфигу_внутри_контейнера}

В нашем случае команда выглядит так:

docker exec -it docker-registry bin/registry garbage-collect /etc/docker/registry/config.yml путь до конфига также может быть следующим: (/var/opt/gitlab/registry/config.yml)

Для чистки места также можно использовать скрипт clean_docker_registry.sh следующего содержания:

#!/bin/sh
REPOPATH=/srv/gitlab/shared/registry/docker/registry/v2/repositories/
echo Docker image name is $1
echo Docker image tag is $2
echo "\n";
TAGPATH=$REPOPATH$1/_manifests/tags/$2/index/sha256
REVPATH=$REPOPATH$1/_manifests/revisions/sha256
for hash in $(ls $TAGPATH -t | tail -n +2)
do
rm -rf $TAGPATH/$hash;
rm -rf $REVPATH/$hash;
done
docker exec -it docker-registry bin/registry garbage-collect /etc/docker/registry/config.yml

Запускаем скрипт с параметрами, первый из которых — имя docker-образа, второй — тэг docker-образа, например:

./clean_docker_registry.sh master/lebed/test latest также можно воспользоваться следующим скриптом:

#!/bin/bash -e
#
# GitLab docker instance registry cleanup script. The idea is to call
# it from a cron job to periodically prune unwanted objects from the
# ever growing docker registry.
#
# Use at your own risks, it can potentially delete a lot of things.
#

# Standard GitLab data path
DATADIR="/var/opt/gitlab"

# Exit with an error message
die () {
    echo "Error: $*" >&2
    exit 1
}

# Display an information message
info () {
    echo "Info: $*"
}

# Remove stuff
remove () {
    rm -rfv -- "$@"
}

##
## Clean a single registry repository
##
clean_repo () {
    local wanted_manifests=()
    local wanted_layers=()
    cd "$1"

    ##
    ## Step #1: remove obsolete manifests, by tag
    ##
    for tagdir in _manifests/tags/*; do
	tag="${tagdir##*/}"

	# It could be that there's no tag
	[ "$tag" = "*" ] && break

	# Preserve the hash of the current image
	current="$(cut -d: -f2 < "$tagdir/current/link")"
	wanted_manifests+=("$current")
	info "tag $tag is at $current"

	# Delete all other manifests
	for i in $tagdir/index/sha256/*; do
	    if [ "$(basename "$i")" = "$current" ]; then
		info "preserving $i"
	    else
		remove "$i"
	    fi
	done
    done

    ##
    ## Step #2: remove obsolete manifests, by revision
    ##
    for revdir in _manifests/revisions/sha256/*; do
	rev="${revdir##*/}"

	# It could be that there's no manifest
	[ "$rev" = "*" ] && break

	# Preserve this manifest?
	found=0
	for i in "${wanted_manifests[@]}"; do
	    [ "$rev" = "$i" ] && found=1
	done

	# Delete all other obsolete manifests
	if [ "$found" = 1 ]; then
	    info "preserving $revdir"
	else
	    remove "$revdir"
	fi
    done

    ##
    ## Step #3: remove obsolete layers
    ##
    for manifest in "${wanted_manifests[@]}"; do

	# Locate the blob data file
	blobfile="$blobdir/$(echo "$manifest" | cut -c -2)/$manifest/data"
	[ -f "$blobfile" ] || error "File $blobfile doesn't exist."

	# Remember all referenced layers (greedy and brutal)
	readarray -t layers < <(grep 'sha256:' < "$blobfile"		\
				    | sed -e 's/"digest": "sha256://'	\
					  -e 's/[ \t]*//'		\
					  -e 's/"//g')
	wanted_layers+=( "${layers[@]}" )
    done

    for layerdir in _layers/sha256/*; do
	layer="${layerdir##*/}"

	# It could be that there's no layer
	[ "$layer" = "*" ] && break

	# Preserve this layer?
	found=0
	for i in "${wanted_layers[@]}"; do
	    [ "$layer" = "$i" ] && found=1
	done

	# Delete all other obsolete layers
	if [ $found = 1 ]; then
	    info "preserving $layerdir"
	else
	    remove "$layerdir"
	fi
    done
}

##
## Entry point
##

# Make sure that the data directory exists
[ -d "$DATADIR" ] || die "Can't find the GitLab data directory"

# Make sure that we have write permissions to it
[ -w "$DATADIR" ] || die "Insufficient permissions to the GitLab data directory"

# Figure out the path to the registry
regdir="$DATADIR/gitlab-rails/shared/registry/docker/registry/v2"
blobdir="$regdir/blobs/sha256"
repodir="$regdir/repositories"
[ -d "$blobdir" ] || die "Can't find the registry blob directory"
[ -d "$repodir" ] || die "Can't find the registry repo directory"

# Lookup all registy repositories
readarray -t repos < <(find "$repodir" -type d								\
			    -exec test -d '{}'/_layers -a -d '{}'/_uploads -a -d '{}'/_manifests \;	\
			    \( -prune -print \))

# Iterate over each repository
for repo in "${repos[@]}"; do
    clean_repo "$repo"
done

# Call the registry garbage collector
exec gitlab-ctl registry-garbage-collect

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

#!/bin/bash -e

# GitLab docker instance registry cleanup script. The idea is to call

# it from a cron job to periodically prune unwanted objects from the

# ever growing docker registry.

# Use at your own risks, it can potentially delete a lot of things.

# Standard GitLab data path

DATADIR="/var/opt/gitlab"

# Exit with an error message

die () {

echo "Error: $*" >&2

exit 1

}

# Display an information message

info () {

echo "Info: $*"

}

# Remove stuff

remove () {

rm -rfv -- "$@"

}

## Clean a single registry repository

clean_repo () {

local wanted_manifests=()

local wanted_layers=()

cd "$1"

## Step #1: remove obsolete manifests, by tag

for tagdir in _manifests/tags/*; do

tag="${tagdir##*/}"

# It could be that there's no tag

[ "$tag" = "*" ] && break

# Preserve the hash of the current image

current="$(cut -d: -f2 < "$tagdir/current/link")"

wanted_manifests+=("$current")

info "tag $tag is at $current"

# Delete all other manifests

for i in $tagdir/index/sha256/*; do

if [ "$(basename "$i")" = "$current" ]; then

info "preserving $i"

else

remove "$i"

done

## Step #2: remove obsolete manifests, by revision

for revdir in _manifests/revisions/sha256/*; do

rev="${revdir##*/}"

# It could be that there's no manifest

[ "$rev" = "*" ] && break

# Preserve this manifest?

found=0

for i in "${wanted_manifests[@]}"; do

[ "$rev" = "$i" ] && found=1

done

# Delete all other obsolete manifests

if [ "$found" = 1 ]; then

info "preserving $revdir"

else

remove "$revdir"

done

## Step #3: remove obsolete layers

for manifest in "${wanted_manifests[@]}"; do

# Locate the blob data file

blobfile="$blobdir/$(echo "$manifest" | cut -c -2)/$manifest/data"

[ -f "$blobfile" ] || error "File $blobfile doesn't exist."

# Remember all referenced layers (greedy and brutal)

readarray -t layers < <(grep 'sha256:' < "$blobfile" \

| sed -e 's/"digest": "sha256://' \

-e 's/[ \t]*//' \

-e 's/"//g')

wanted_layers+=( "${layers[@]}" )

done

for layerdir in _layers/sha256/*; do

layer="${layerdir##*/}"

# It could be that there's no layer

[ "$layer" = "*" ] && break

# Preserve this layer?

found=0

for i in "${wanted_layers[@]}"; do

[ "$layer" = "$i" ] && found=1

done

# Delete all other obsolete layers

if [ $found = 1 ]; then

info "preserving $layerdir"

else

remove "$layerdir"

done

}

## Entry point

# Make sure that the data directory exists

[ -d "$DATADIR" ] || die "Can't find the GitLab data directory"

# Make sure that we have write permissions to it

[ -w "$DATADIR" ] || die "Insufficient permissions to the GitLab data directory"

# Figure out the path to the registry

regdir="$DATADIR/gitlab-rails/shared/registry/docker/registry/v2"

blobdir="$regdir/blobs/sha256"

repodir="$regdir/repositories"

[ -d "$blobdir" ] || die "Can't find the registry blob directory"

[ -d "$repodir" ] || die "Can't find the registry repo directory"

# Lookup all registy repositories

readarray -t repos < <(find "$repodir" -type d \

-exec test -d '{}'/_layers -a -d '{}'/_uploads -a -d '{}'/_manifests \; \

$ -prune -print $)

# Iterate over each repository

for repo in "${repos[@]}"; do

clean_repo "$repo"

done

# Call the registry garbage collector

exec gitlab-ctl registry-garbage-collect

========================================================================================

Рабочий скрипт для запуска которого необходимо
/root/scripts/gitlab-registry-gc.sh -k 5 -r
k 5 - это количество оставляемых слоёв:

ну или как в самом скрипте указано:

{ echo "STARTING EXEC $(date)"; /root/clean.sh -k 5 -r; } >> /root/stdout_gitlab-registry-gc.$(date "+\%Y\%m").log 2>&1

#!/bin/bash

# This is a modification of gitlab-gc.sh script created by Peter Bábics (pbabics/gitlab-gc.sh)

# Improvements
#   - Searching in all BASE_PATH, not fixing the search to a depth of 2
#   - Directories without valid tags or revisions directories won't be processed (to avoid unexpected issues)
#   - Logging in case there's nothing to delete
#   - running registry-garbage-collect only when something has been deleted

# crontab example
# Run everynight the script, keeping 5 old revisions and running registry-garbage-collector at the end. All stdout logs sent to a file rotated every month
# 00 23 * * * { echo "STARTING EXEC $(date)"; /root/scripts/gitlab-registry-gc.sh -k 5 -r; } >> /root/scripts/logs/stdout_gitlab-registry-gc.$(date "+\%Y\%m").log 2>&1

BASE_PATH=/var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories

DRY_RUN=0
KEEP_LAST_IMAGES=10
RUN_GARBAGE_COLLECTOR=0
GITLAB_CTL_COMMAND=`which gitlab-ctl`


if [ ! -x "${GITLAB_CTL_COMMAND}" ]; then
    echo "Missing gitlab-ctl command"
    exit 1
fi


while (( "$#" )); do
    case "$1" in

        "-b" | "--base-path")
            BASE_PATH=$2
            shift
            ;;

        "-r" | "--run-gc")
            RUN_GARBAGE_COLLECTOR=1
            ;;

        "-d"|"--dry-run")
            DRY_RUN=1
            ;;

        "-k"|"--keep")
            if ! ( echo $2 | grep -q '^[0-9]\+$') || [ $2 -eq 0 ]; then
                echo "Invalid value for keep last images '$2'"
                exit 1
            fi
            KEEP_LAST_IMAGES=$2
            shift
            ;;

        "-h"|"--help")
            echo "Usage: ${0} [options]"
            echo "Options:"
            echo -e "\t-k NUM, --keep NUM"
            echo -e "\t\tKeeps last NUM revisions, except current tags"
            echo
            echo -e "\t-d, --dry-run"
            echo -e "\t\tEnables dry run, no changes will be made"
            echo
            echo -e "\t-b, --base-path"
            echo -e "\t\tSets base path of Gitlab Registry repository storage"
            echo
            echo -e "\t-r, --run-gc"
            echo -e "\t\tStarts garbage collector after revision removal"
            exit 0
            ;;

        *)
            echo "Unknown argument: $1"
            exit 1
            ;;
    esac
    shift
done

IFS=$'\n'
used_hashes=`mktemp`
marked_hashes=`mktemp`
found=""

#for repository in `find ${BASE_PATH} -mindepth 2 -maxdepth 2 -type d | sed "s#${BASE_PATH}/##"`; do
for repository in `find ${BASE_PATH} -name "_manifests" -type d | sed -e "s#${BASE_PATH}/##" -e "s#/_manifests##"`; do
    echo "# Processing repository: $repository"
    ls ${BASE_PATH}/${repository}/_manifests/tags/*/current/link >/dev/null 2>&1 || { echo "ERROR: Invalid repository. No tags/*/current/link files found"; echo; continue; }
    test -d "${BASE_PATH}/${repository}/_manifests/revisions/sha256" || { echo "ERROR: Invalid repository. revisions/sha256 dir not found"; echo; continue; }

    for tag_hash in ${BASE_PATH}/${repository}/_manifests/tags/*/current/link; do 
        cat "${tag_hash}" | cut -d':' -f2; 
    done > "${used_hashes}"
    
    echo "# Removing revisions of $repository:"
    ls -t ${BASE_PATH}/${repository}/_manifests/revisions/sha256 | fgrep -vf "${used_hashes}" | tail -n+${KEEP_LAST_IMAGES} | tee ${marked_hashes}
    if test -s $marked_hashes ; then
      if [ ${DRY_RUN} -ne 1 ]; then
          cat ${marked_hashes} | sed "s#^#${BASE_PATH}/${repository}/_manifests/revisions/sha256/#" | xargs rm -rf && found="true"
      fi
    else
      echo "Nothing to delete"
    fi
    echo
done
rm ${used_hashes}
rm ${marked_hashes}


if [ ${DRY_RUN} -eq 0 -a ${RUN_GARBAGE_COLLECTOR} -eq 1 ]; then
    if test "$found"; then
      "${GITLAB_CTL_COMMAND}" registry-garbage-collect
    else
      echo "Skipping execution of ${GITLAB_CTL_COMMAND} registry-garbage-collect because nothing was deleted"
    fi
fi

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

#!/bin/bash

# This is a modification of gitlab-gc.sh script created by Peter Bábics (pbabics/gitlab-gc.sh)

# Improvements

# - Searching in all BASE_PATH, not fixing the search to a depth of 2

# - Directories without valid tags or revisions directories won't be processed (to avoid unexpected issues)

# - Logging in case there's nothing to delete

# - running registry-garbage-collect only when something has been deleted

# crontab example

# Run everynight the script, keeping 5 old revisions and running registry-garbage-collector at the end. All stdout logs sent to a file rotated every month

# 00 23 * * * { echo "STARTING EXEC $(date)"; /root/scripts/gitlab-registry-gc.sh -k 5 -r; } >> /root/scripts/logs/stdout_gitlab-registry-gc.$(date "+\%Y\%m").log 2>&1

BASE_PATH=/var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories

DRY_RUN=0

KEEP_LAST_IMAGES=10

RUN_GARBAGE_COLLECTOR=0

GITLAB_CTL_COMMAND=`which gitlab-ctl`

if [ ! -x "${GITLAB_CTL_COMMAND}" ]; then

echo "Missing gitlab-ctl command"

exit 1

while (( "$#" )); do

case "$1" in

"-b" | "--base-path")

BASE_PATH=$2

shift

;;

"-r" | "--run-gc")

RUN_GARBAGE_COLLECTOR=1

;;

"-d"|"--dry-run")

DRY_RUN=1

;;

"-k"|"--keep")

if ! ( echo $2 | grep -q '^[0-9]\+$') || [ $2 -eq 0 ]; then

echo "Invalid value for keep last images '$2'"

exit 1

KEEP_LAST_IMAGES=$2

shift

;;

"-h"|"--help")

echo "Usage: ${0} [options]"

echo "Options:"

echo -e "\t-k NUM, --keep NUM"

echo -e "\t\tKeeps last NUM revisions, except current tags"

echo

echo -e "\t-d, --dry-run"

echo -e "\t\tEnables dry run, no changes will be made"

echo

echo -e "\t-b, --base-path"

echo -e "\t\tSets base path of Gitlab Registry repository storage"

echo

echo -e "\t-r, --run-gc"

echo -e "\t\tStarts garbage collector after revision removal"

exit 0

;;

echo "Unknown argument: $1"

exit 1

;;

esac

shift

done

IFS=$'\n'

used_hashes=`mktemp`

marked_hashes=`mktemp`

found=""

#for repository in `find ${BASE_PATH} -mindepth 2 -maxdepth 2 -type d | sed "s#${BASE_PATH}/##"`; do

for repository in `find ${BASE_PATH} -name "_manifests" -type d | sed -e "s#${BASE_PATH}/##" -e "s#/_manifests##"`; do

echo "# Processing repository: $repository"

ls ${BASE_PATH}/${repository}/_manifests/tags/*/current/link >/dev/null 2>&1 || { echo "ERROR: Invalid repository. No tags/*/current/link files found"; echo; continue; }

test -d "${BASE_PATH}/${repository}/_manifests/revisions/sha256" || { echo "ERROR: Invalid repository. revisions/sha256 dir not found"; echo; continue; }

for tag_hash in ${BASE_PATH}/${repository}/_manifests/tags/*/current/link; do

cat "${tag_hash}" | cut -d':' -f2;

done > "${used_hashes}"

echo "# Removing revisions of $repository:"

ls -t ${BASE_PATH}/${repository}/_manifests/revisions/sha256 | fgrep -vf "${used_hashes}" | tail -n+${KEEP_LAST_IMAGES} | tee ${marked_hashes}

if test -s $marked_hashes ; then

if [ ${DRY_RUN} -ne 1 ]; then

cat ${marked_hashes} | sed "s#^#${BASE_PATH}/${repository}/_manifests/revisions/sha256/#" | xargs rm -rf && found="true"

else

echo "Nothing to delete"

echo

done

rm ${used_hashes}

rm ${marked_hashes}

if [ ${DRY_RUN} -eq 0 -a ${RUN_GARBAGE_COLLECTOR} -eq 1 ]; then

if test "$found"; then

"${GITLAB_CTL_COMMAND}" registry-garbage-collect

else

echo "Skipping execution of ${GITLAB_CTL_COMMAND} registry-garbage-collect because nothing was deleted"

Centos

Очистка места в gitlab-registry

https://github.com/midnight47/

Сентябрь 2024
Пн	Вт	Ср	Чт	Пт	Сб	Вс
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30