GitLab 迁移升级

May 27, 2022 4 min read

GitLab 迁移升级

step 1 、配置 docker-compose

mkdir gitlab-compose

cat << EOF >>gitlab-compose/docker-compose.yaml 
version: '3.3'
services:
  gitlab-ce:
    image: gitlab/gitlab-ce:8.14.0-ce.0
    container_name: gitlab-ce
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://192.168.166.166:18080'
        gitlab_rails['gitlab_shell_ssh_port'] = 12222
    ports:
      - "443:443"
      - "18080:18080"
      - "12222:22"
    restart: "no"
    networks:
      bridge:
        ipv4_address: 172.32.32.166
    volumes:
      - '/data/gitlab-ce/config:/etc/gitlab'
      - '/data/gitlab-ce/logs:/var/log/gitlab'
      - '/data/gitlab-ce/data:/var/opt/gitlab'

networks:
  bridge:
    driver: bridge
    ipam:
      config:
      - subnet: 172.32.32.1/24
EOF

# 启动 gitlab-ce
cd gitlab-compose
docker-compose up -d

step 2 、导入数据

docker exec -it gitlab-ce bash

# 修改数据库配置，二进制包安装的使用了 bitnami 和 postgres 两个 role ，
# 官方 docker 版本不存在这两个 role ，避免产生未知影响，将这两个 role 手动创建出来。
su - gitlab-psql

psql -d gitlabhq_production -h /var/opt/gitlab/postgresql

gitlabhq_production=# .\du
                                 List of roles
     Role name     |                   Attributes                   | Member of 
-------------------+------------------------------------------------+-----------
 gitlab            |                                                | {}
 gitlab-psql       | Superuser, Create role, Create DB, Replication | {}
 gitlab_replicator | Replication                                    | {}

gitlabhq_production=# create role bitnami;
CREATE ROLE
gitlabhq_production=# create role postgres;
CREATE ROLE
gitlabhq_production=# alter role bitnami superuser;
ALTER ROLE
gitlabhq_production=# alter role bitnami login;
ALTER ROLE
gitlabhq_production=# alter role postgres login;
ALTER ROLE
gitlabhq_production=# alter role postgres superuser;


gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq

gitlab-rake gitlab:backup:restore BACKUP=1653564453 

Do you want to continue (yes/no)? yes
Removing all tables. Press `Ctrl-C` within 5 seconds to abort
Cleaning the database ... 
done
Restoring database ... 

gitlab-ctl start unicorn
gitlab-ctl start sidekiq

external_url 参数修改成 ip + port 的是否注意也要修改 docker 启动时映射的端口，如 external_url = 10.0.0.2:8080 ，dockers run -p 8080:8080

具体问题参考官方文档 https://docs.gitlab.com/ee/install/docker.html

step 3 、 升级 gitlab

升级必须按照官方指定的升级路径进行升级，具体升级路径如下：

Upgrade paths

Upgrading across multiple GitLab versions in one go is only possible by accepting downtime. The following examples assume downtime is acceptable while upgrading. If you don’t want any downtime, read how to upgrade with zero downtime.

Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the version-specific upgrade instructions:

8.11.Z -> 8.12.0 -> 8.17.7 -> 9.5.10 -> 10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.8.8 -> 13.12.15 -> 14.0.12 -> 14.9.0 -> latest 14.Y.Z

按照路径重复执行

  • 1、docker-compose down
  • 2、 修改 docker-compose.yaml 文件中具体的镜像版本
  • 3、docker-compose up -d

升级过程中可能存在问题，主要是权限问题，可根据实际问题进行修复。

Permission problems

When updating from older GitLab Docker images you might encounter permission problems. This happens when users in previous images were not preserved correctly. There’s script that fixes permissions for all files.

To fix your container, execute update-permissions and restart the container afterwards:

sudo docker exec gitlab update-permissions
sudo docker restart gitlab

Linux ACL issues

If you are using file ACLs on the Docker host, the docker group requires full access to the volumes in order for GitLab to work:

getfacl $GITLAB_HOME

# file: $GITLAB_HOME
# owner: XXXX
# group: XXXX
user::rwx
group::rwx
group:docker:rwx
mask::rwx
default:user::rwx
default:group::rwx
default:group:docker:rwx
default😷:rwx
default:other::r-x

If these are not correct, set them with:

sudo setfacl -mR default:group:docker:rwx $GITLAB_HOME

The default group is docker. If you changed the group, be sure to update your commands

gitlab 13.12.15 升级到最新稳定版本

官方升级路径：13.12.15 -> 14.0.12 -> 14.3.6 -> 14.9.5 -> 14.10.Z -> 15.0.Z -> 15.4.0 -> latest 15.Y.Z，Z 代表最新的小版本，目前 Z 为 5。

开启维护模式

gitlab-rails c
::Gitlab::CurrentSettings.update!(maintenance_mode: true)
::Gitlab::CurrentSettings.update!(maintenance_mode_message: "GitLab 升级中，请稍微再试")

迁移过程

tips: 13.12 升到 14.0 的过程时， gitlab 不再支持 legacy storage 需要先将后端存储转为 hashed storage。

In GitLab 13.0, hashed storage is enabled by default and the legacy storage is deprecated. GitLab 14.0 eliminates support for legacy storage. If you’re on GitLab 13.0 and later, switching new projects to legacy storage is not possible. The option to choose between hashed and legacy storage in the Admin Area has been disabled

Checking for unmigrated data on legacy storage

Legacy storage is no longer supported. Please migrate your data to hashed storage.

# To have a summary and then a list of projects and their attachments using legacy storage
# Projects
gitlab-rake gitlab:storage:legacy_projects
gitlab-rake gitlab:storage:list_legacy_projects

# Attachments
gitlab-rake gitlab:storage:legacy_attachments
gitlab-rake gitlab:storage:list_legacy_attachments

# To have a summary and then a list of projects and their attachments using hashed storage
# Projects
gitlab-rake gitlab:storage:hashed_projects
gitlab-rake gitlab:storage:list_hashed_projects

# Attachments
gitlab-rake gitlab:storage:hashed_attachments
gitlab-rake gitlab:storage:list_hashed_attachments

# This task must be run on any machine that has Rails/Sidekiq configured, and the task schedules all your existing projects and attachments associated with it to be migrated to the Hashed storage type
gitlab-rake gitlab:storage:migrate_to_hashed
# 等待后台任务完成  Main menu > Admin > Monitoring > Background Jobs. Watch how long the hashed_storage:hashed_storage_project_migrate queue takes to finish.

# 回滚
gitlab-rake gitlab:storage:rollback_to_legacy

故障排查：

  • readonly 仓库
root@29d0c185b57f:/# gitlab-rake gitlab:storage:migrate_to_hashed
Enqueuing migration of 113 projects in batches of 200. Done!
# 迁移过程中会出现任务未完成现象
root@29d0c185b57f:/# gitlab-rake gitlab:storage:list_legacy_projects
* Found 7 projects using Legacy Storage
  - data/gateway (id: 223)
    data/gateway
  ...

root@29d0c185b57f:/# gitlab-rake gitlab:storage:legacy_projects
* Found 7 projects using Legacy Storage

# 根据官方解释 'If you have set projects `read only` they might fail to migrate.'
# 可以检查上述 7 个项目是否为 `read only` 状态
# 1、Start a Rails console.
gitlab-rails console
# 2、Check if the project is read only
project = Project.find_by_full_path('data/gateway')
project.repository_read_only
# 3、If it returns true (not nil or false), set it writable:
project.update!(repository_read_only: false)

# 4、Re-run the migration Rake task.
gitlab-rake gitlab:storage:migrate_to_hashed
# 5、Set the project read-only again:
project.update!(repository_read_only: true)

wget -O /tmp/fix-legacy-hashed-storage-migration.rb https://gitlab.com/snippets/2039252/raw
gitlab-rails runner /tmp/fix-legacy-hashed-storage-migration.rb
gitlab-rake gitlab:storage:migrate_to_hashed
gitlab-rake gitlab:storage:list_legacy_projects
gitlab-rake gitlab:storage:list_legacy_attachments

再次确认检查：

root@29d0c185b57f:/# gitlab-rake gitlab:storage:legacy_projects
* Found 0 projects using Legacy Storage
root@29d0c185b57f:/# gitlab-rake gitlab:storage:legacy_attachments
* Found 0 attachments using Legacy Storage

上述结果表明，迁移存储完成。

Gitlab 14 以后的版本采用 Batched background migrations 方式迁移。

To update database tables in batches, GitLab can use batched background migrations. These migrations are created by GitLab developers and run automatically on upgrade. However, such migrations are limited in scope to help with migrating some integer database columns to bigint. This is needed to prevent integer overflow for some tables.

升级前需要确定 background migrations 状态

All migrations must have a [Finished status](https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#manually-finishing-a-batched-background-migration) before you upgrade GitLab. You can check the status of existing migrations

# pending migrations
gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count'
# Failed migrations
# For GitLab 14.0-14.9:
gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count'
# For GitLab 14.10 and later
gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count'

# Check the status of batched background migrations
# 检查数据库
gitlab-psql
gitlabhq_production=# select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
            job_class_name             |     table_name     | column_name |                                   job_arguments                                    
---------------------------------------+--------------------+-------------+------------------------------------------------------------------------------------
 CopyColumnUsingBackgroundMigrationJob | ci_job_artifacts   | id          | [["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob | ci_builds          | id          | [["id", "stage_id"], ["id_convert_to_bigint", "stage_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob | taggings           | id          | [["id", "taggable_id"], ["id_convert_to_bigint", "taggable_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob | ci_stages          | id          | [["id"], ["id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob | ci_builds_metadata | id          | [["id"], ["id_convert_to_bigint"]]
# 未完成之前升级会遇到如下错误
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':


Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run
      [execute] rake aborted!
                StandardError: An error has occurred, all later migrations canceled:
                
                Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
                
                Finalize it manualy by running
                
                	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
                
                For more information, check the documentation
                
                	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished

# 后台任务卡住处理
# What do you do if your background migrations are stuck
gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
gitlab-rails c
# Execute the following in the rails console
scheduled_queue = Sidekiq::ScheduledSet.new
pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq
pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }

开启或关闭 batched background migrations

# enable
Feature.enable(:execute_batched_migrations_on_schedule)
# disable
Feature.disable(:execute_batched_migrations_on_schedule)
  • step 3 按照升级路径依次升级镜像

重要 升级前确定后台迁移完成之后在升级 管理中心 > 监控 > 后台迁移

# 依次修改 docker-compose.yml 中的镜像： 3.12.15 -> 14.0.12 -> 14.3.6 -> 14.9.5 -> 14.10.Z -> 15.0.Z -> 15.4.0 -> latest 15.Y.Z
# 更新
docker-compose up -d
gitlab
Infee Fang
Infee Fang
互联网二手搬砖工