hashicorp vault в режиме HA с integrated storage(raft)

Thank you for reading this post, don't forget to subscribe!

1.Общее по hashicorp vault
2.Установка keepalived
3.Установка hashicorp vault
3.1Создание ключей
3.2.Конфигурация vault
3.3.Добавление сертификатов в доверенные
4.Создание метода аутентификации user/password
4.1Создание admin policy для user
4.2.Добавление group с admin правами
4.3.Добавление дополнительного пользователя в группу
4.4 Настроим все те же доступы но с помощью команд
5.Аутентификация через gitlab
6. Auto unseal hashicorp vault
6.1 установка minikub
6.2 подготовка хранилища для PV (nfs)
6.3 установка vault в minikub
6.4 auto unseal vault с помощью другого vault
7.Бэкап данных(backup raft)
8. Настройка LDAP — freeipa
9. Добавляем секреты из vault в POD (vault установлен в k8s)
9.1 secrets-store-csi-driver
9.2 Vault Secrets Operator (наиболее оптимальный вариант для использования)
9.2.1 Пример с подкидыванием секрета в deployment
9.2.2 Включение Vault Secrets Operator reloader

10.SSH аутентификация через vault
10.1 SSH аутентификация через vault OTP (временный пароль)
10.2 SSH аутентификация через vault SSH (certificate authority)

Vault позволяет хранить секреты, токены, пароли, сертификаты и т.п., предоставляя к ним ограниченный и безопасный доступ. Естественно, внутри Vault все хранится в зашифрованном виде.
Под катом процедура инсталляция и запуск Vault на трех узлах в отказоустойчивом режиме.

В простом варианте Vault состоит из некоторого количества взаимосвязанных между собой серверов, один из которых работает в режиме active, а остальные в режиме standby.

HA, либо High Availability позволяет выдержать потерю одного или нескольких узлов (в зависимости от кофигурации), сохранив при этом доступ к данным. Не со всеми бэкэндами, используемыми для хранения, возможно использовать HA механизм.

Для хранения данных Vault может использовать большое количество бэкэндов.

Полностью поддерживаемые HC бэкэнды (не все, но основные):
Filesystem – название говорит само за себя. Храним данные на локальной файловой системе, там же, где и установлен Vault. Не подходит для HA конфигурации;

Consul – Так же является разработкой HashiCorp. Использование в качестве бэкэнда позволяет реализовать Vault HA. Однако, данный вариант потребует развертывания дополнительных машин и организации отдельного кластера Consul (ну либо все в одном, но я не любитель такого);

Integrated Storage (Raft) – данные Vault так же хранятся на файловой системе, но при этом реплицируются на другие узлы кластера, позволяя им оставаться доступными даже в случае выхода из строя одного из узлов. Данный вариант поддерживает функционал HA.

Мой выбор в данном случае – Integrated Storage.

Итак, начнем с подготовки машин. В моем случае это три машины (минимальное количество для кворума)

добавляем в hosts на все тачки

cat /etc/hosts

192.168.1.171 vault1 vault1.test.local
192.168.1.172 vault2 vault2.test.local
192.168.1.173 vault3 vault3.test.local
192.168.1.174 vault.test.local

айпишник 192.168.1.174 - виртуальный

ставим на все тачки:

yum install -y yum-utils wget curl
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo

и отключаем SELINUX на ВСЕХ тачках

cat /etc/selinux/config

SELINUX=disabled

1	SELINUX=disabled

sed -i 's|SELINUX=enforcing|SELINUX=disabled|g' /etc/selinux/config

ребутаемся.

теперь поставим keepalived ОБЯЗАТЕЛЬНО 2 версии

на каждую ноду ставим:

wget http://www.nosuchhost.net/~cheese/fedora/packages/epel-7/x86_64/cheese-release-7-1.noarch.rpm
rpm -Uvh cheese-release*rpm
yum install keepalived -y

systemctl enable keepalived

если есть firewall то добавляем в разрешённые протокол vrrp

firewall-cmd --permanent --add-rich-rule='rule protocol value="vrrp" accept'
firewall-cmd --reload

mkdir /usr/libexec/keepalived/

cat <<"EOF" | sudo tee /usr/libexec/keepalived/vault-health > /dev/null
#!/bin/bash
if [ $# -ne 1 ];then
 echo "WARNING: You must provide health check URL"
 exit 1
else
 CHECK_URL=$1
 CMD=$(/usr/bin/curl -k -I ${CHECK_URL} 2>/dev/null | grep "200" | wc -l)
 if [ ${CMD} -eq 1 ];then
   exit 0
 else
   exit 1
 fi
fi
EOF

cat <<"EOF" | sudo tee /usr/libexec/keepalived/vault-health > /dev/null

#!/bin/bash

if [ $# -ne 1 ];then

echo "WARNING: You must provide health check URL"

exit 1

else

CHECK_URL=$1

CMD=$(/usr/bin/curl -k -I ${CHECK_URL} 2>/dev/null | grep "200" | wc -l)

if [ ${CMD} -eq 1 ];then

exit 0

else

exit 1

EOF

chmod +x /usr/libexec/keepalived/vault-health

mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old

cat /etc/keepalived/keepalived.conf

# Global definitions configuration block
global_defs {
    router_id LVS_LB
}

vrrp_script check_vault_health {
    script "/usr/libexec/keepalived/vault-health https://localhost:8200/v1/sys/health"
    interval 1
    timeout 3
    rise 3
    fall 3
}

vrrp_instance VI_1 {
    state MASTER
    virtual_router_id 100
    priority 100
    advert_int 1

    # Please make sure to verify interface name using 'nmcli connection show'
    interface enp0s3

    virtual_ipaddress {
        # Floating IP
        192.168.1.174 dev enp0s3 label enp0s3:vip
    }

    track_interface {
        # Please make sure to verify interface name using 'nmcli connection show'
        enp0s3
    }

    track_script {
        check_vault_health
    }
}

# Global definitions configuration block

global_defs {

router_id LVS_LB

}

vrrp_script check_vault_health {

script "/usr/libexec/keepalived/vault-health https://localhost:8200/v1/sys/health"

interval 1

timeout 3

rise 3

fall 3

}

vrrp_instance VI_1 {

state MASTER

virtual_router_id 100

priority 100

advert_int 1

# Please make sure to verify interface name using 'nmcli connection show'

interface enp0s3

virtual_ipaddress {

# Floating IP

192.168.1.174 dev enp0s3 label enp0s3:vip

}

track_interface {

# Please make sure to verify interface name using 'nmcli connection show'

enp0s3

}

track_script {

check_vault_health

}

groupadd -r keepalived_script
useradd -r -s /sbin/nologin -g keepalived_script -M keepalived_script
systemctl enable --now keepalived.service

Далее ставим vault

yum -y install vault

проверяем:

[root@vault1 ~]# vault version
Vault v1.11.3 (17250b25303c6418c283c95b1d5a9c9f16174fe8), built 2022-08-26T10:27:10Z

[root@vault1 ~]# vault version

Vault v1.11.3 (17250b25303c6418c283c95b1d5a9c9f16174fe8), built 2022-08-26T10:27:10Z

Открываем необходимые для работы Vault порты:

firewall-cmd --add-port=8200/tcp --permanent
firewall-cmd --add-port=8201/tcp --permanent
firewall-cmd –reload

firewall-cmd --add-port=8200/tcp --permanent

firewall-cmd --add-port=8201/tcp --permanent

firewall-cmd –reload

Далее необходимо обзавестись сертификатами. Если таковые отсутствуют – выпустить свои, самоподписанные. Все операции я выполняю с первой ноды.

Сперва выпустим свой CA сертификат, а затем с его помощью подпишем сертификаты для всех узлов кластера:

cd /opt/vault/tls/
openssl genrsa 2048 > vault-ca-key.pem
openssl req -new -x509 -nodes -days 3650 -key vault-ca-key.pem -out vault-ca-cert.pem

cd /opt/vault/tls/

openssl genrsa 2048 > vault-ca-key.pem

openssl req -new -x509 -nodes -days 3650 -key vault-ca-key.pem -out vault-ca-cert.pem

Country Name (2 letter code) [XX]:
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:test.local
Email Address []:

Country Name (2 letter code) [XX]:

State or Province Name (full name) []:

Locality Name (eg, city) [Default City]:

Organization Name (eg, company) [Default Company Ltd]:

Organizational Unit Name (eg, section) []:

Common Name (eg, your name or your server's hostname) []:test.local

Email Address []:

Теперь подготовим конфигурационные файлы, содержащие Subject Alternate Name (SAN) для каждого из узлов. Важно, чтобы в SAN был корректней хостнейм и IP каждого узла:

echo "[v3_ca]
subjectAltName = @alt_names
[alt_names]
DNS.1 = vault1.test.local
IP.1 = 192.168.1.171
IP.2 = 127.0.0.1
" > ./cert-a.cfg

echo "[v3_ca]

subjectAltName = @alt_names

[alt_names]

DNS.1 = vault1.test.local

IP.1 = 192.168.1.171

IP.2 = 127.0.0.1

" > ./cert-a.cfg

echo "[v3_ca]
subjectAltName = @alt_names
[alt_names]
DNS.1 = vault2.test.local
IP.1 = 192.168.1.172
IP.2 = 127.0.0.1
" > ./cert-b.cfg

echo "[v3_ca]

subjectAltName = @alt_names

[alt_names]

DNS.1 = vault2.test.local

IP.1 = 192.168.1.172

IP.2 = 127.0.0.1

" > ./cert-b.cfg

echo "[v3_ca]
subjectAltName = @alt_names
[alt_names]
DNS.1 = vault3.test.local
IP.1 = 192.168.1.173
IP.2 = 127.0.0.1
" > ./cert-c.cfg

echo "[v3_ca]

subjectAltName = @alt_names

[alt_names]

DNS.1 = vault3.test.local

IP.1 = 192.168.1.173

IP.2 = 127.0.0.1

" > ./cert-c.cfg

Теперь для каждого из узлов сформируем CSR файл:

openssl req -newkey rsa:2048 -nodes -keyout vault-a-key.pem -out vault-a-csr.pem -subj "/CN=vault1.test.local"
openssl req -newkey rsa:2048 -nodes -keyout vault-b-key.pem -out vault-b-csr.pem -subj "/CN=vault2.test.local"
openssl req -newkey rsa:2048 -nodes -keyout vault-c-key.pem -out vault-c-csr.pem -subj "/CN=vault3.test.local"

openssl req -newkey rsa:2048 -nodes -keyout vault-a-key.pem -out vault-a-csr.pem -subj "/CN=vault1.test.local"

openssl req -newkey rsa:2048 -nodes -keyout vault-b-key.pem -out vault-b-csr.pem -subj "/CN=vault2.test.local"

openssl req -newkey rsa:2048 -nodes -keyout vault-c-key.pem -out vault-c-csr.pem -subj "/CN=vault3.test.local"

И выпустим сертификаты на основании запросов:

openssl x509 -req -set_serial 01 -days 3650 -in vault-a-csr.pem -out vault-a-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-a.cfg
openssl x509 -req -set_serial 01 -days 3650 -in vault-b-csr.pem -out vault-b-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-b.cfg
openssl x509 -req -set_serial 01 -days 3650 -in vault-c-csr.pem -out vault-c-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-c.cfg

openssl x509 -req -set_serial 01 -days 3650 -in vault-a-csr.pem -out vault-a-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-a.cfg

openssl x509 -req -set_serial 01 -days 3650 -in vault-b-csr.pem -out vault-b-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-b.cfg

openssl x509 -req -set_serial 01 -days 3650 -in vault-c-csr.pem -out vault-c-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-c.cfg

вывод следующий:

[root@vault1 tls]# openssl x509 -req -set_serial 01 -days 3650 -in vault-a-csr.pem -out vault-a-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-a.cfg
Signature ok
subject=/CN=vault1.test.local
Getting CA Private Key

[root@vault1 tls]# openssl x509 -req -set_serial 01 -days 3650 -in vault-b-csr.pem -out vault-b-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-b.cfg
Signature ok
subject=/CN=vault2.test.local
Getting CA Private Key

[root@vault1 tls]# openssl x509 -req -set_serial 01 -days 3650 -in vault-c-csr.pem -out vault-c-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-c.cfg
Signature ok
subject=/CN=vault3.test.local
Getting CA Private Key

[root@vault1 tls]# openssl x509 -req -set_serial 01 -days 3650 -in vault-a-csr.pem -out vault-a-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-a.cfg

Signature ok

subject=/CN=vault1.test.local

Getting CA Private Key

[root@vault1 tls]# openssl x509 -req -set_serial 01 -days 3650 -in vault-b-csr.pem -out vault-b-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-b.cfg

Signature ok

subject=/CN=vault2.test.local

Getting CA Private Key

[root@vault1 tls]# openssl x509 -req -set_serial 01 -days 3650 -in vault-c-csr.pem -out vault-c-cert.pem -CA vault-ca-cert.pem -CAkey vault-ca-key.pem -extensions v3_ca -extfile ./cert-c.cfg

Signature ok

subject=/CN=vault3.test.local

Getting CA Private Key

Скопируем сертификаты и ключи на узлы B и C:

scp ./vault-b-key.pem ./vault-b-cert.pem ./vault-ca-cert.pem vault2.test.local:/opt/vault/tls
scp ./vault-c-key.pem ./vault-c-cert.pem ./vault-ca-cert.pem vault3.test.local:/opt/vault/tls

1 2	scp ./vault-b-key.pem ./vault-b-cert.pem ./vault-ca-cert.pem vault2.test.local:/opt/vault/tls scp ./vault-c-key.pem ./vault-c-cert.pem ./vault-ca-cert.pem vault3.test.local:/opt/vault/tls

На каждом из узлов установим соответствующие права для доступа к файлам сертификатов и ключам:

chown root:root /opt/vault/tls/vault-*-cert.pem /opt/vault/tls/vault-ca-cert.pem
chown root:vault /opt/vault/tls/vault-*-key.pem
chmod 0644 /opt/vault/tls/vault-*-cert.pem /opt/vault/tls/vault-ca-cert.pem
chmod 0640 /opt/vault/tls/vault-*-key.pem

chown root:root /opt/vault/tls/vault-*-cert.pem /opt/vault/tls/vault-ca-cert.pem

chown root:vault /opt/vault/tls/vault-*-key.pem

chmod 0644 /opt/vault/tls/vault-*-cert.pem /opt/vault/tls/vault-ca-cert.pem

chmod 0640 /opt/vault/tls/vault-*-key.pem

Теперь, когда для каждой из нод готовы сертификаты.

Перейдем к конфигурации Vault.
Отредактируем конфигурационный файл vault для первой ноды:

cp /etc/vault.d/vault.hcl /etc/vault.d/vault.hcl.backup
vim /etc/vault.d/vault.hcl

1 2	cp /etc/vault.d/vault.hcl /etc/vault.d/vault.hcl.backup vim /etc/vault.d/vault.hcl

cluster_addr  = "https://192.168.1.171:8201"
api_addr      = "https://192.168.1.171:8200"
disable_mlock = true

ui = true

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_ca_cert_file   = "/opt/vault/tls/vault-ca-cert.pem"
  tls_cert_file      = "/opt/vault/tls/vault-a-cert.pem"
  tls_key_file       = "/opt/vault/tls/vault-a-key.pem"

}

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault1.test.local"

  retry_join {
    leader_tls_servername   = "vault1.test.local"
    leader_api_addr         = "https://192.168.1.171:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-a-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-a-key.pem"
  }
  retry_join {
    leader_tls_servername   = "vault2.test.local"
    leader_api_addr         = "https://192.168.1.172:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-a-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-a-key.pem"
  }
  retry_join {
    leader_tls_servername   = "vault3.test.local"
    leader_api_addr         = "https://192.168.1.173:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-a-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-a-key.pem"
  }
}

cluster_addr = "https://192.168.1.171:8201"

api_addr = "https://192.168.1.171:8200"

disable_mlock = true

ui = true

listener "tcp" {

address = "0.0.0.0:8200"

tls_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

tls_cert_file = "/opt/vault/tls/vault-a-cert.pem"

tls_key_file = "/opt/vault/tls/vault-a-key.pem"

}

storage "raft" {

path = "/opt/vault/data"

node_id = "vault1.test.local"

retry_join {

leader_tls_servername = "vault1.test.local"

leader_api_addr = "https://192.168.1.171:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-a-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-a-key.pem"

}

retry_join {

leader_tls_servername = "vault2.test.local"

leader_api_addr = "https://192.168.1.172:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-a-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-a-key.pem"

}

retry_join {

leader_tls_servername = "vault3.test.local"

leader_api_addr = "https://192.168.1.173:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-a-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-a-key.pem"

}

вторая нода:

cp /etc/vault.d/vault.hcl /etc/vault.d/vault.hcl.backup

[root@vault2 ~]# cat /etc/vault.d/vault.hcl

cluster_addr  = "https://192.168.1.172:8201"
api_addr      = "https://192.168.1.172:8200"
disable_mlock = true

ui = true

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_ca_cert_file   = "/opt/vault/tls/vault-ca-cert.pem"
  tls_cert_file      = "/opt/vault/tls/vault-b-cert.pem"
  tls_key_file       = "/opt/vault/tls/vault-b-key.pem"

}

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault2.test.local"

  retry_join {
    leader_tls_servername   = "vault1.test.local"
    leader_api_addr         = "https://192.168.1.171:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-b-key.pem"
  }
  retry_join {
    leader_tls_servername   = "vault2.test.local"
    leader_api_addr         = "https://192.168.1.172:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-b-key.pem"
  }
  retry_join {
    leader_tls_servername   = "vault3.test.local"
    leader_api_addr         = "https://192.168.1.173:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-b-key.pem"
  }
}

cluster_addr = "https://192.168.1.172:8201"

api_addr = "https://192.168.1.172:8200"

disable_mlock = true

ui = true

listener "tcp" {

address = "0.0.0.0:8200"

tls_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

tls_cert_file = "/opt/vault/tls/vault-b-cert.pem"

tls_key_file = "/opt/vault/tls/vault-b-key.pem"

}

storage "raft" {

path = "/opt/vault/data"

node_id = "vault2.test.local"

retry_join {

leader_tls_servername = "vault1.test.local"

leader_api_addr = "https://192.168.1.171:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-b-key.pem"

}

retry_join {

leader_tls_servername = "vault2.test.local"

leader_api_addr = "https://192.168.1.172:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-b-key.pem"

}

retry_join {

leader_tls_servername = "vault3.test.local"

leader_api_addr = "https://192.168.1.173:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-b-key.pem"

}

третья нода:

cp /etc/vault.d/vault.hcl /etc/vault.d/vault.hcl.backup

[root@vault3 ~]# cat /etc/vault.d/vault.hcl

cluster_addr  = "https://192.168.1.173:8201"
api_addr      = "https://192.168.1.173:8200"
disable_mlock = true

ui = true

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_ca_cert_file   = "/opt/vault/tls/vault-ca-cert.pem"
  tls_cert_file      = "/opt/vault/tls/vault-c-cert.pem"
  tls_key_file       = "/opt/vault/tls/vault-c-key.pem"

}

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault3.test.local"

  retry_join {
    leader_tls_servername   = "vault1.test.local"
    leader_api_addr         = "https://192.168.1.171:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-c-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-c-key.pem"
  }
  retry_join {
    leader_tls_servername   = "vault2.test.local"
    leader_api_addr         = "https://192.168.1.172:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-c-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-c-key.pem"
  }
  retry_join {
    leader_tls_servername   = "vault3.test.local"
    leader_api_addr         = "https://192.168.1.173:8200"
    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"
    leader_client_cert_file = "/opt/vault/tls/vault-c-cert.pem"
    leader_client_key_file  = "/opt/vault/tls/vault-c-key.pem"
  }
}

cluster_addr = "https://192.168.1.173:8201"

api_addr = "https://192.168.1.173:8200"

disable_mlock = true

ui = true

listener "tcp" {

address = "0.0.0.0:8200"

tls_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

tls_cert_file = "/opt/vault/tls/vault-c-cert.pem"

tls_key_file = "/opt/vault/tls/vault-c-key.pem"

}

storage "raft" {

path = "/opt/vault/data"

node_id = "vault3.test.local"

retry_join {

leader_tls_servername = "vault1.test.local"

leader_api_addr = "https://192.168.1.171:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-c-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-c-key.pem"

}

retry_join {

leader_tls_servername = "vault2.test.local"

leader_api_addr = "https://192.168.1.172:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-c-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-c-key.pem"

}

retry_join {

leader_tls_servername = "vault3.test.local"

leader_api_addr = "https://192.168.1.173:8200"

leader_ca_cert_file = "/opt/vault/tls/vault-ca-cert.pem"

leader_client_cert_file = "/opt/vault/tls/vault-c-cert.pem"

leader_client_key_file = "/opt/vault/tls/vault-c-key.pem"

}

Обратите внимание как меняются поля с адресами, сертификатами, а также node_id.

api_addr – адрес и порт, на котором будет доступен API сервер;
cluster_addr – адрес и порт по которому будут взаимодействовать кластерные сервисы;
disable_mlock – рекомендуемый параметр при использовании Integrated Storage;
ui – включение доступа к веб-интерфейсу Vault;
в секции listener указываются сертификаты, которые будут использованы при сетевом взаимодействии. У каждой ноды они свои, за исключением CA, данный сертификат одинаковый для всех.

Секцию storage стоит так же рассмотреть:
path = “/opt/vault/data” – директория, где будут храниться данные Vault;
node_id = “vault-a.vmik.lab” – id, с которым нода будет участвовать в кластере. У каждой ноды он должен отличаться;
Далее идут несколько секций retry_join, с перечислением всех узлов кластера. Поскольку доподлинно неизвестно, какой из узлов будет активным при запуске служб Vault, будет произведена попытка подключения к каждому из узлов.
Здесь же указываются адреса узлов – leader_api_addr, leader_tls_servername – хостнейм сервера, который должен совпадать с тем, что прописано в сертификате данного сервера. Так же указываются сертификаты, которыми клиент будет подключаться к лидеру (у каждой из нод свои сертификаты, которые мы создавали ранее).

Теперь, когда конфигурационные файлы готовы, возвращаемся на первый узел. Добавляем Vault в автозагрузку и автоматически его запускаем:

[root@vault1 tls]# systemctl enable --now vault.service
[root@vault1 tls]# systemctl status vault.service
● vault.service - "HashiCorp Vault - A tool for managing secrets"
   Loaded: loaded (/usr/lib/systemd/system/vault.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-09-04 12:20:56 +06; 16s ago
     Docs: https://www.vaultproject.io/docs/
 Main PID: 1641 (vault)
   CGroup: /system.slice/vault.service
           └─1641 /usr/bin/vault server -config=/etc/vault.d/vault.hcl

Sep 04 12:21:12 vault1 vault[1641]: 2022-09-04T12:21:12.987+0600 [ERROR] core: failed to get raft challenge: leader_addr=https://192.168.1.172:8200 error="erro...on refused"
Sep 04 12:21:12 vault1 vault[1641]: 2022-09-04T12:21:12.988+0600 [ERROR] core: failed to get raft challenge: leader_addr=https://192.168.1.171:8200
Sep 04 12:21:12 vault1 vault[1641]: error=
Sep 04 12:21:12 vault1 vault[1641]: | error during raft bootstrap init call: Error making API request.
Sep 04 12:21:12 vault1 vault[1641]: |
Sep 04 12:21:12 vault1 vault[1641]: | URL: PUT https://192.168.1.171:8200/v1/sys/storage/raft/bootstrap/challenge
Sep 04 12:21:12 vault1 vault[1641]: | Code: 503. Errors:
Sep 04 12:21:12 vault1 vault[1641]: |
Sep 04 12:21:12 vault1 vault[1641]: | * Vault is sealed
Sep 04 12:21:12 vault1 vault[1641]: 2022-09-04T12:21:12.988+0600 [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
Hint: Some lines were ellipsized, use -l to show in full.

[root@vault1 tls]# systemctl enable --now vault.service

[root@vault1 tls]# systemctl status vault.service

● vault.service - "HashiCorp Vault - A tool for managing secrets"

Loaded: loaded (/usr/lib/systemd/system/vault.service; enabled; vendor preset: disabled)

Active: active (running) since Sun 2022-09-04 12:20:56 +06; 16s ago

Docs: https://www.vaultproject.io/docs/

Main PID: 1641 (vault)

CGroup: /system.slice/vault.service

└─1641 /usr/bin/vault server -config=/etc/vault.d/vault.hcl

Sep 04 12:21:12 vault1 vault[1641]: 2022-09-04T12:21:12.987+0600 [ERROR] core: failed to get raft challenge: leader_addr=https://192.168.1.172:8200 error="erro…on refused"

Sep 04 12:21:12 vault1 vault[1641]: 2022-09-04T12:21:12.988+0600 [ERROR] core: failed to get raft challenge: leader_addr=https://192.168.1.171:8200

Sep 04 12:21:12 vault1 vault[1641]: error=

Sep 04 12:21:12 vault1 vault[1641]: | error during raft bootstrap init call: Error making API request.

Sep 04 12:21:12 vault1 vault[1641]: |

Sep 04 12:21:12 vault1 vault[1641]: | URL: PUT https://192.168.1.171:8200/v1/sys/storage/raft/bootstrap/challenge

Sep 04 12:21:12 vault1 vault[1641]: | Code: 503. Errors:

Sep 04 12:21:12 vault1 vault[1641]: |

Sep 04 12:21:12 vault1 vault[1641]: | * Vault is sealed

Sep 04 12:21:12 vault1 vault[1641]: 2022-09-04T12:21:12.988+0600 [ERROR] core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"

Hint: Some lines were ellipsized, use -l to show in full.

Проверим статус Vault, предварительно отменив проверку сертификатов, выставив значение специальной переменной:

[root@vault1 tls]# export VAULT_SKIP_VERIFY="true"
[root@vault1 tls]# vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            1.11.3
Build Date         2022-08-26T10:27:10Z
Storage Type       raft
HA Enabled         true

[root@vault1 tls]# export VAULT_SKIP_VERIFY="true"

[root@vault1 tls]# vault status

Key Value

--- -----

Seal Type shamir

Initialized false

Sealed true

Total Shares 0

Threshold 0

Unseal Progress 0/0

Unseal Nonce n/a

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

HA Enabled true

Как можно заметить на текущий момент Vault не инициализирован (Initialized false), а также запечатан (Sealed true).

Примечание: добавив наш CA файл, сгенерированный ранее, в список доверенных сертификатов, прибегать к отмене проверки сертификатов будет необязательно.

Инициализируем Vault:

vault operator init

[root@vault1 tls]# vault operator init
Unseal Key 1: vQhtGnKfP6z8ZhH/3LbxaffPzb5FEmEsV/U1MDyHo7mr
Unseal Key 2: QebWSJkXEzmFwMA8aYhOYgDohRjIRJW/dVJP9UyYL25z
Unseal Key 3: GpnuWi5gTWtAxIkxFXSU69hLUlgSEBTfj6heEn5ZB1vj
Unseal Key 4: 9D4UWacP8ysqPDVCgLIBGbl7yp9m35dawvwbH5KtiWAd
Unseal Key 5: 5fUt/8zz8/CfongdTRuNASiAxYa+cbsSDNdKajw2/aGc

Initial Root Token: hvs.yInppPkxqfAcVcODMVlvLjN3

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

[root@vault1 tls]# vault operator init

Unseal Key 1: vQhtGnKfP6z8ZhH/3LbxaffPzb5FEmEsV/U1MDyHo7mr

Unseal Key 2: QebWSJkXEzmFwMA8aYhOYgDohRjIRJW/dVJP9UyYL25z

Unseal Key 3: GpnuWi5gTWtAxIkxFXSU69hLUlgSEBTfj6heEn5ZB1vj

Unseal Key 4: 9D4UWacP8ysqPDVCgLIBGbl7yp9m35dawvwbH5KtiWAd

Unseal Key 5: 5fUt/8zz8/CfongdTRuNASiAxYa+cbsSDNdKajw2/aGc

Initial Root Token: hvs.yInppPkxqfAcVcODMVlvLjN3

Vault initialized with 5 key shares and a key threshold of 3. Please securely

distribute the key shares printed above. When the Vault is re-sealed,

restarted, or stopped, you must supply at least 3 of these keys to unseal it

before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to

reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of

existing unseal keys shares. See "vault operator rekey" for more information.

Vault выдаст 5 ключей, которые необходимо использовать при «распечатке». Как сказано выше, после каждой остановки либо перезапуска, Vault будет вновь находиться в запечатанном состоянии и для открытия нужно будет использовать любые три из пяти предоставленных ключей. Терять эти ключи не стоит!
Так же нам предоставляется Root Token для доступа к Vault с максимальными правами.

Распечатаем Vault. Операцию vault operator unseal нужно будет провести 3 раза. Каждый раз с разным ключом:

[root@vault1 tls]# vault operator unseal
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       04777048-4ba3-1574-a858-f8728011541c
Version            1.11.3
Build Date         2022-08-26T10:27:10Z
Storage Type       raft
HA Enabled         true
[root@vault1 tls]# vault operator unseal
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       04777048-4ba3-1574-a858-f8728011541c
Version            1.11.3
Build Date         2022-08-26T10:27:10Z
Storage Type       raft
HA Enabled         true
[root@vault1 tls]# vault operator unseal
Unseal Key (will be hidden):
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.11.3
Build Date              2022-08-26T10:27:10Z
Storage Type            raft
Cluster Name            vault-cluster-bf7b95a3
Cluster ID              6aa2ee81-da82-bbc8-1f02-789051c5ec44
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    31
Raft Applied Index      31

[root@vault1 tls]# vault operator unseal

Unseal Key (will be hidden):

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed true

Total Shares 5

Threshold 3

Unseal Progress 1/3

Unseal Nonce 04777048-4ba3-1574-a858-f8728011541c

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

HA Enabled true

[root@vault1 tls]# vault operator unseal

Unseal Key (will be hidden):

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed true

Total Shares 5

Threshold 3

Unseal Progress 2/3

Unseal Nonce 04777048-4ba3-1574-a858-f8728011541c

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

HA Enabled true

[root@vault1 tls]# vault operator unseal

Unseal Key (will be hidden):

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed false

Total Shares 5

Threshold 3

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

Cluster Name vault-cluster-bf7b95a3

Cluster ID 6aa2ee81-da82-bbc8-1f02-789051c5ec44

HA Enabled true

HA Cluster n/a

HA Mode standby

Active Node Address <none>

Raft Committed Index 31

Raft Applied Index 31

На данном этапе мы настроили одну ноду Vault, а также инициализировали и распечатали ее. Теперь подключим в кластер две других ноды.
Ранее мы уже подготовили и разместили конфигурационные файлы на нодах B и C.
Подключаемся к ноде B, запускаем Vault:

systemctl enable --now vault
systemctl status vault

Теперь, в отличии от первой ноды, инициализировать Vault больше не нужно, но новая нода все еще в запечатанном состоянии. Распечатаем ноду B:

[root@vault2 ~]# export VAULT_SKIP_VERIFY="true"

[root@vault2 ~]# vault operator unseal
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       f53d3c54-f4e4-902f-ba11-a4c826c2feeb
Version            1.11.3
Build Date         2022-08-26T10:27:10Z
Storage Type       raft
HA Enabled         true

[root@vault2 ~]# vault operator unseal
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       f53d3c54-f4e4-902f-ba11-a4c826c2feeb
Version            1.11.3
Build Date         2022-08-26T10:27:10Z
Storage Type       raft
HA Enabled         true

[root@vault2 ~]# vault operator unseal
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.11.3
Build Date         2022-08-26T10:27:10Z
Storage Type       raft
HA Enabled         true

[root@vault2 ~]# vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.11.3
Build Date              2022-08-26T10:27:10Z
Storage Type            raft
Cluster Name            vault-cluster-bf7b95a3
Cluster ID              6aa2ee81-da82-bbc8-1f02-789051c5ec44
HA Enabled              true
HA Cluster              https://192.168.1.171:8201
HA Mode                 standby
Active Node Address     https://192.168.1.171:8200
Raft Committed Index    37
Raft Applied Index      37

[root@vault2 ~]# export VAULT_SKIP_VERIFY="true"

[root@vault2 ~]# vault operator unseal

Unseal Key (will be hidden):

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed true

Total Shares 5

Threshold 3

Unseal Progress 1/3

Unseal Nonce f53d3c54-f4e4-902f-ba11-a4c826c2feeb

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

HA Enabled true

[root@vault2 ~]# vault operator unseal

Unseal Key (will be hidden):

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed true

Total Shares 5

Threshold 3

Unseal Progress 2/3

Unseal Nonce f53d3c54-f4e4-902f-ba11-a4c826c2feeb

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

HA Enabled true

[root@vault2 ~]# vault operator unseal

Unseal Key (will be hidden):

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed true

Total Shares 5

Threshold 3

Unseal Progress 0/3

Unseal Nonce n/a

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

HA Enabled true

[root@vault2 ~]# vault status

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed false

Total Shares 5

Threshold 3

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

Cluster Name vault-cluster-bf7b95a3

Cluster ID 6aa2ee81-da82-bbc8-1f02-789051c5ec44

HA Enabled true

HA Cluster https://192.168.1.171:8201

HA Mode standby

Active Node Address https://192.168.1.171:8200

Raft Committed Index 37

Raft Applied Index 37

После распечатки нода будет подключена к кластеру. Можно обратить внимание, что новая нода находится в режиме standby, так же можно определить текущий адрес активной ноды. Это нода A.

Подключимся к ноде C и выполним аналогичные действия:

[root@vault3 ~]# systemctl enable --now vault
[root@vault3 ~]# export VAULT_SKIP_VERIFY="true"

[root@vault3 ~]# vault operator unseal
[root@vault3 ~]# vault operator unseal
[root@vault3 ~]# vault operator unseal

[root@vault3 ~]# vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.11.3
Build Date              2022-08-26T10:27:10Z
Storage Type            raft
Cluster Name            vault-cluster-bf7b95a3
Cluster ID              6aa2ee81-da82-bbc8-1f02-789051c5ec44
HA Enabled              true
HA Cluster              https://192.168.1.171:8201
HA Mode                 standby
Active Node Address     https://192.168.1.171:8200
Raft Committed Index    42
Raft Applied Index      42

[root@vault3 ~]# systemctl enable --now vault

[root@vault3 ~]# export VAULT_SKIP_VERIFY="true"

[root@vault3 ~]# vault operator unseal

[root@vault3 ~]# vault status

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed false

Total Shares 5

Threshold 3

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

Cluster Name vault-cluster-bf7b95a3

Cluster ID 6aa2ee81-da82-bbc8-1f02-789051c5ec44

HA Enabled true

HA Cluster https://192.168.1.171:8201

HA Mode standby

Active Node Address https://192.168.1.171:8200

Raft Committed Index 42

Raft Applied Index 42

Теперь мы успешно сформировали кластер из трех нод. Вернемся на первую и проверим состояние кластера.

Для начала авторизуемся с токеном, который был получен ранее:

[root@vault1 tls]# vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.yInppPkxqfAcVcODMVlvLjN3
token_accessor       hmOVGjYCG7nTELrcoyxEeoxL
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

[root@vault1 tls]# vault login

Token (will be hidden):

Success! You are now authenticated. The token information displayed below

is already stored in the token helper. You do NOT need to run "vault login"

again. Future Vault requests will automatically use this token.

Key Value

--- -----

token hvs.yInppPkxqfAcVcODMVlvLjN3

token_accessor hmOVGjYCG7nTELrcoyxEeoxL

token_duration ∞

token_renewable false

token_policies ["root"]

identity_policies []

policies ["root"]

И проверим статус хранилища:

[root@vault1 tls]# vault operator raft list-peers
Node                 Address               State       Voter
----                 -------               -----       -----
vault1.test.local    192.168.1.171:8201    leader      true
vault2.test.local    192.168.1.172:8201    follower    true
vault3.test.local    192.168.1.173:8201    follower    true

[root@vault1 tls]# vault operator raft list-peers

Node Address State Voter

---- ------- ----- -----

vault1.test.local 192.168.1.171:8201 leader true

vault2.test.local 192.168.1.172:8201 follower true

vault3.test.local 192.168.1.173:8201 follower true

По статусу видно 3 сервера, один из которых в статусе лидера, а два других – ведомые.

Попробуем добавить новый секрет в хранилище. Для этого используем механизм kv (key-value):

[root@vault1 tls]# vault secrets enable -path=test-secrets/ kv
Success! Enabled the kv secrets engine at: test-secrets/

1 2	[root@vault1 tls]# vault secrets enable -path=test-secrets/ kv Success! Enabled the kv secrets engine at: test-secrets/

И поместим секрет под именем db:

[root@vault1 tls]# vault kv put test-secrets/db password=123456789
Success! Data written to: test-secrets/db

1 2	[root@vault1 tls]# vault kv put test-secrets/db password=123456789 Success! Data written to: test-secrets/db

Посмотрим список всех секретов в нашем KV хранилище test-secrets:

[root@vault1 tls]# vault kv list test-secrets
Keys
----
db

[root@vault1 tls]# vault kv list test-secrets

Keys

----

А также значение секрета db:

[root@vault1 tls]# vault kv get test-secrets/db
====== Data ======
Key         Value
---         -----
password    123456789

[root@vault1 tls]# vault kv get test-secrets/db

====== Data ======

Key Value

--- -----

password 123456789

Проверим работоспособность механизма HA. Отключим ноду A:

[root@vault1 tls]# poweroff

Подключимся к ноде B и проверим статус:

[root@vault2 ~]# vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.11.3
Build Date              2022-08-26T10:27:10Z
Storage Type            raft
Cluster Name            vault-cluster-bf7b95a3
Cluster ID              6aa2ee81-da82-bbc8-1f02-789051c5ec44
HA Enabled              true
HA Cluster              https://192.168.1.172:8201
HA Mode                 active
Active Since            2022-09-04T06:37:35.678401666Z
Raft Committed Index    50
Raft Applied Index      50

[root@vault2 ~]# vault status

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed false

Total Shares 5

Threshold 3

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

Cluster Name vault-cluster-bf7b95a3

Cluster ID 6aa2ee81-da82-bbc8-1f02-789051c5ec44

HA Enabled true

HA Cluster https://192.168.1.172:8201

HA Mode active

Active Since 2022-09-04T06:37:35.678401666Z

Raft Committed Index 50

Raft Applied Index 50

HA Mode изменено со standby на active. Запросим список узлов raft:

но сначала передадим в переменные окружения наш токен рутовый, мы его получили когда инициировали кластер:

[root@vault2 ~]# export VAULT_TOKEN="hvs.yInppPkxqfAcVcODMVlvLjN3"

[root@vault2 ~]# vault operator raft list-peers
Node                 Address               State       Voter
----                 -------               -----       -----
vault1.test.local    192.168.1.171:8201    follower    true
vault2.test.local    192.168.1.172:8201    leader      true
vault3.test.local    192.168.1.173:8201    follower    true

[root@vault2 ~]# export VAULT_TOKEN="hvs.yInppPkxqfAcVcODMVlvLjN3"

[root@vault2 ~]# vault operator raft list-peers

Node Address State Voter

---- ------- ----- -----

vault1.test.local 192.168.1.171:8201 follower true

vault2.test.local 192.168.1.172:8201 leader true

vault3.test.local 192.168.1.173:8201 follower true

Лидер так же был перенесен на ноду B. В завершении запросим ранее созданный секрет:

[root@vault2 ~]# vault kv get test-secrets/db
====== Data ======
Key         Value
---         -----
password    123456789

[root@vault2 ~]# vault kv get test-secrets/db

====== Data ======

Key Value

--- -----

password 123456789

Vault продолжает функционировать при потере одной активной ноды. Отключим ноду B. Теперь из трех узлов доступен только один. Проверим работоспособность:

[root@vault2 ~]# poweroff

Статус Vault с ноды C:

[root@vault3 ~]# vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.11.3
Build Date              2022-08-26T10:27:10Z
Storage Type            raft
Cluster Name            vault-cluster-bf7b95a3
Cluster ID              6aa2ee81-da82-bbc8-1f02-789051c5ec44
HA Enabled              true
HA Cluster              https://192.168.1.172:8201
HA Mode                 standby
Active Node Address     https://192.168.1.172:8200
Raft Committed Index    53
Raft Applied Index      52

[root@vault3 ~]# vault status

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed false

Total Shares 5

Threshold 3

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

Cluster Name vault-cluster-bf7b95a3

Cluster ID 6aa2ee81-da82-bbc8-1f02-789051c5ec44

HA Enabled true

HA Cluster https://192.168.1.172:8201

HA Mode standby

Active Node Address https://192.168.1.172:8200

Raft Committed Index 53

Raft Applied Index 52

Активная нода не изменилась. Нода C все так же в режиме standby. Аналогично, не удастся запросить и секрет, поскольку запрос перенаправляется на ранее активную ноду (B):

[root@vault3 ~]# vault kv get test-secrets/db
Get "https://192.168.1.172:8200/v1/sys/internal/ui/mounts/test-secrets/db": dial tcp 192.168.1.172:8200: connect: no route to host

[root@vault3 ~]# vault kv get test-secrets/db

Get "https://192.168.1.172:8200/v1/sys/internal/ui/mounts/test-secrets/db": dial tcp 192.168.1.172:8200: connect: no route to host

Запустим обратно ноду A. После запуска, как и ожидается, нода A находится в статусе sealed:

[root@vault1 ~]# export VAULT_SKIP_VERIFY="true"
[root@vault1 ~]# vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.11.3
Build Date         2022-08-26T10:27:10Z
Storage Type       raft
HA Enabled         true

[root@vault1 ~]# export VAULT_SKIP_VERIFY="true"

[root@vault1 ~]# vault status

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed true

Total Shares 5

Threshold 3

Unseal Progress 0/3

Unseal Nonce n/a

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

HA Enabled true

Распечатаем:

[root@vault1 ~]# vault operator unseal
[root@vault1 ~]# vault operator unseal
[root@vault1 ~]# vault operator unseal

[root@vault1 ~]# vault operator unseal

проверим:

[root@vault1 ~]# vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.11.3
Build Date              2022-08-26T10:27:10Z
Storage Type            raft
Cluster Name            vault-cluster-bf7b95a3
Cluster ID              6aa2ee81-da82-bbc8-1f02-789051c5ec44
HA Enabled              true
HA Cluster              https://192.168.1.173:8201
HA Mode                 standby
Active Node Address     https://192.168.1.173:8200
Raft Committed Index    58
Raft Applied Index      58

[root@vault1 ~]# vault status

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed false

Total Shares 5

Threshold 3

Version 1.11.3

Build Date 2022-08-26T10:27:10Z

Storage Type raft

Cluster Name vault-cluster-bf7b95a3

Cluster ID 6aa2ee81-da82-bbc8-1f02-789051c5ec44

HA Enabled true

HA Cluster https://192.168.1.173:8201

HA Mode standby

Active Node Address https://192.168.1.173:8200

Raft Committed Index 58

Raft Applied Index 58

Работа кластера восстановлена. Нода С теперь активная. Доступ к секретам так же восстановлен:

[root@vault3 ~]# export VAULT_TOKEN="hvs.yInppPkxqfAcVcODMVlvLjN3"
[root@vault3 ~]# vault kv get test-secrets/db
====== Data ======
Key         Value
---         -----
password    123456789

[root@vault3 ~]# export VAULT_TOKEN="hvs.yInppPkxqfAcVcODMVlvLjN3"

[root@vault3 ~]# vault kv get test-secrets/db

====== Data ======

Key Value

--- -----

password 123456789

Запустим обратно выключенную ноду B и распечатаем. На этом работа по настройке кластера Vault закончена.

Нерассмотренным остался один вопрос – как подключаться внешним клиентам? Подключение к любой из standby нод перенаправит запрос на active ноду, поэтому знать текущий адрес активной ноды не обязательно.
Однако, клиент может не знать все адреса Vault и в случае отключения известной ноды, доступ клиента к Vault может прекратиться.

Наиболее приемлемые на мой взгляд решения:
1. Внешний балансировщик нагрузки между узлами Vault с заранее известным и постоянным адресом. HAProxy, либо nginx;
2. Общий IP-адрес между узлами кластера Vault, например, на основе keepalived.

Добавим сертификаты в доверенные, чтобы каждый раз не закидывать в переменные окружения

export VAULT_SKIP_VERIFY="true"

для этого на каждой из нод выполним:

[root@vault1 ~]# update-ca-trust enable
[root@vault1 ~]# cp /opt/vault/tls/vault-ca-cert.pem /etc/pki/ca-trust/source/anchors/
[root@vault1 ~]# update-ca-trust extract

[root@vault2 ~]# update-ca-trust enable
[root@vault2 ~]# cp /opt/vault/tls/vault-ca-cert.pem /etc/pki/ca-trust/source/anchors/
[root@vault2 ~]# update-ca-trust extract

[root@vault3 ~]# update-ca-trust enable
[root@vault3 ~]# cp /opt/vault/tls/vault-ca-cert.pem /etc/pki/ca-trust/source/anchors/
[root@vault3 ~]# update-ca-trust extract

[root@vault1 ~]# update-ca-trust enable

[root@vault1 ~]# cp /opt/vault/tls/vault-ca-cert.pem /etc/pki/ca-trust/source/anchors/

[root@vault1 ~]# update-ca-trust extract

[root@vault2 ~]# update-ca-trust enable

[root@vault2 ~]# cp /opt/vault/tls/vault-ca-cert.pem /etc/pki/ca-trust/source/anchors/

[root@vault2 ~]# update-ca-trust extract

[root@vault3 ~]# update-ca-trust enable

[root@vault3 ~]# cp /opt/vault/tls/vault-ca-cert.pem /etc/pki/ca-trust/source/anchors/

[root@vault3 ~]# update-ca-trust extract

Подключимся к кластеру:

https://vault.test.local:8200/

аутентифицируемся с помощью root token

Initial Root Token: hvs.yInppPkxqfAcVcODMVlvLjN3

видим созданный нами ранее секрет

теперь создадим метод аутентификации по user/password

создадим теперь пользователя

зайдём под этим пользователем и проверим что нам доступно:

как видим недостаточно доступов.

теперь создадим admin policy пример возьмём отсюда:

https://learn.hashicorp.com/tutorials/vault/policies

# Read system health check
path "sys/health"
{
  capabilities = ["read", "sudo"]
}

# Create and manage ACL policies broadly across Vault

# List existing policies
path "sys/policies/acl"
{
  capabilities = ["list"]
}

# Create and manage ACL policies
path "sys/policies/acl/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Enable and manage authentication methods broadly across Vault

# Manage auth methods broadly across Vault
path "auth/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Create, update, and delete auth methods
path "sys/auth/*"
{
  capabilities = ["create", "update", "delete", "sudo"]
}

# List auth methods
path "sys/auth"
{
  capabilities = ["read"]
}

# Enable and manage the key/value secrets engine at `secret/` path

# List, create, update, and delete key/value secrets
path "secret/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage secrets engines
path "sys/mounts/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List existing secrets engines.
path "sys/mounts"
{
  capabilities = ["read"]
}

# Read system health check

path "sys/health"

{

capabilities = ["read", "sudo"]

}

# Create and manage ACL policies broadly across Vault

# List existing policies

path "sys/policies/acl"

{

capabilities = ["list"]

}

# Create and manage ACL policies

path "sys/policies/acl/*"

{

capabilities = ["create", "read", "update", "delete", "list", "sudo"]

}

# Enable and manage authentication methods broadly across Vault

# Manage auth methods broadly across Vault

path "auth/*"

{

capabilities = ["create", "read", "update", "delete", "list", "sudo"]

}

# Create, update, and delete auth methods

path "sys/auth/*"

{

capabilities = ["create", "update", "delete", "sudo"]

}

# List auth methods

path "sys/auth"

{

capabilities = ["read"]

}

# Enable and manage the key/value secrets engine at `secret/` path

# List, create, update, and delete key/value secrets

path "secret/*"

{

capabilities = ["create", "read", "update", "delete", "list", "sudo"]

}

# Manage secrets engines

path "sys/mounts/*"

{

capabilities = ["create", "read", "update", "delete", "list", "sudo"]

}

# List existing secrets engines.

path "sys/mounts"

{

capabilities = ["read"]

}

policy создали, теперь нужно создать Entities (сущности)

логинимся под нашим пользователем и проверям какие теперь есть доступы:

как видим теперь мы видим все наши секреты возможность создавать новые policy тоже появилась, попробуем создать новый секрет:

как видим секрет создался.

====================================================

Создание группы с админскими правами

создаём сущность Entities и alias для неё. alias должен совпадать с именем нашего пользователя

"entity" - это абстрактное представление человека или сервиса, который взаимодействует с сервером Vault. Entity может быть связан с одним или несколькими методами аутентификации, такими как userpass, ldap или github, и к нему можно прикрепить политики для управления доступом к ресурсам в Vault.

При использовании метода аутентификации userpass в Vault для каждого пользователя, зарегистрированного с этим методом, создается entity. Entity хранит информацию о пользователе, такую как имя пользователя, пароль и любые политики, связанные с ним. При входе пользователя с помощью userpass Vault использует информацию, хранящуюся в entity, чтобы аутентифицировать пользователя и определить, к каким ресурсам у него есть доступ.

отметим что в данном случае мы НЕ ПРИВЯЗЫВАЕМ policy

теперь создаём alias

теперь создаём группу(в текущем примере internal):

и уже тут цепляем нашу policy

проверяем нашу группу:

id в members группы:

это id Entities

заходим под нашим пользователем mid и проверяем что всё ок и доступов на добавление секретов хватает:

создадим ещё пользователя и добавим в ту же группу:

теперь добавим его в нашу группу, для этого создаём Entities

добавляем теперь в группу:

отмечу что все эти действия я выполнял уже под пользователем MID а не root

всё, можно теперь логиниться под этим пользователем(mid2).

Повторим те же действия, но с помощью команд:

Создадим userpass - доступ по логину и паролю:

YAML

vault auth enable userpass

1

vault auth enable userpass

YAML

[root@vault1 ~]# vault auth enable userpass Success! Enabled userpass auth method at: userpass/

1
2

[root@vault1 ~]# vault auth enable userpass
Success! Enabled userpass auth method at: userpass/

Создаём policy с доступа admin

echo 'path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}' | vault policy write admin -

echo 'path "*" {

capabilities = ["create", "read", "update", "delete", "list", "sudo"]

}' | vault policy write admin -

[root@vault1 ~]# echo 'path "*" {
>   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
> }' | vault policy write admin -
Success! Uploaded policy: admin

[root@vault1 ~]# echo 'path "*" {

> capabilities = ["create", "read", "update", "delete", "list", "sudo"]

> }' | vault policy write admin -

Success! Uploaded policy: admin

Создаём пользователя mid с авторизацией по логину и паролю:

YAML

vault write auth/userpass/users/mid password=123456789

1

vault write auth/userpass/users/mid password=123456789

YAML

[root@vault1 ~]# vault write auth/userpass/users/mid password=123456789 Success! Data written to: auth/userpass/users/mid

1
2
3

[root@vault1 ~]# vault write auth/userpass/users/mid password=123456789
Success! Data written to: auth/userpass/users/mid

создаём entity для пользователя mid

vault write identity/entity name="mid"

1	vault write identity/entity name="mid"

результат:

[root@vault1 ~]# vault write identity/entity name="mid"
Key        Value
---        -----
aliases    <nil>
id         98cb3fe3-6aac-8c02-7ea1-a576baf676f6
name       mid

[root@vault1 ~]# vault write identity/entity name="mid"

Key Value

--- -----

aliases <nil>

id 98cb3fe3-6aac-8c02-7ea1-a576baf676f6

name mid

если забыли id его можно посмотреть вот так:

vault read identity/entity/name/mid

1	vault read identity/entity/name/mid

результат:

[root@vault1 ~]# vault read identity/entity/name/mid
Key                    Value
---                    -----
aliases                []
creation_time          2023-01-09T14:34:20.638988226Z
direct_group_ids       []
disabled               false
group_ids              []
id                     98cb3fe3-6aac-8c02-7ea1-a576baf676f6
inherited_group_ids    []
last_update_time       2023-01-09T14:34:20.638988226Z
merged_entity_ids      <nil>
metadata               <nil>
name                   mid
namespace_id           root
policies               []

[root@vault1 ~]# vault read identity/entity/name/mid

Key Value

--- -----

aliases []

creation_time 2023-01-09T14:34:20.638988226Z

direct_group_ids []

disabled false

group_ids []

id 98cb3fe3-6aac-8c02-7ea1-a576baf676f6

inherited_group_ids []

last_update_time 2023-01-09T14:34:20.638988226Z

merged_entity_ids <nil>

metadata <nil>

name mid

namespace_id root

policies []

ещё нам нужна будет переменная mount_accessor
чтобы увидеть mount_accessor выполним команду:

vault auth list

[root@vault1 ~]# vault auth list
Path         Type        Accessor                  Description                Version
----         ----        --------                  -----------                -------
token/       token       auth_token_4ad2bc9d       token based credentials    n/a
userpass/    userpass    auth_userpass_8fabd83b    n/a                        n/a

vault auth list

[root@vault1 ~]# vault auth list

Path Type Accessor Description Version

---- ---- -------- ----------- -------

token/ token auth_token_4ad2bc9d token based credentials n/a

userpass/ userpass auth_userpass_8fabd83b n/a n/a

всё, мы получили значение для mount_accessor и canonical_id теперь можно выполнять команду на создание alias для entity:

 vault write identity/entity-alias name="mid" canonical_id=<canonical_id> mount_accessor=<Accessor>

1	vault write identity/entity-alias name="mid" canonical_id=<canonical_id> mount_accessor=<Accessor>

результат:

[root@vault1 ~]# vault write identity/entity-alias name="mid" canonical_id=98cb3fe3-6aac-8c02-7ea1-a576baf676f6 mount_accessor=auth_userpass_8fabd83b
Key             Value
---             -----
canonical_id    98cb3fe3-6aac-8c02-7ea1-a576baf676f6
id              9f1bfdf3-a461-2933-2a36-b136815bd32a

[root@vault1 ~]# vault write identity/entity-alias name="mid" canonical_id=98cb3fe3-6aac-8c02-7ea1-a576baf676f6 mount_accessor=auth_userpass_8fabd83b

Key Value

--- -----

canonical_id 98cb3fe3-6aac-8c02-7ea1-a576baf676f6

id 9f1bfdf3-a461-2933-2a36-b136815bd32a

если забыли id то его можно посмотреть вот так(тут будут все ключи):

[root@vault1 ~]# vault list identity/entity-alias/id/
Keys
----
9f1bfdf3-a461-2933-2a36-b136815bd32a

[root@vault1 ~]# vault list identity/entity-alias/id/

Keys

----

9f1bfdf3-a461-2933-2a36-b136815bd32a

получить информацию по нему можно так:

[root@vault1 ~]# vault read identity/entity-alias/id/9f1bfdf3-a461-2933-2a36-b136815bd32a
Key                          Value
---                          -----
canonical_id                 98cb3fe3-6aac-8c02-7ea1-a576baf676f6
creation_time                2023-01-09T14:36:04.149468544Z
custom_metadata              map[]
id                           9f1bfdf3-a461-2933-2a36-b136815bd32a
last_update_time             2023-01-09T14:36:04.149468544Z
local                        false
merged_from_canonical_ids    <nil>
metadata                     <nil>
mount_accessor               auth_userpass_8fabd83b
mount_path                   auth/userpass/
mount_type                   userpass
name                         mid
namespace_id                 root

[root@vault1 ~]# vault read identity/entity-alias/id/9f1bfdf3-a461-2933-2a36-b136815bd32a

Key Value

--- -----

canonical_id 98cb3fe3-6aac-8c02-7ea1-a576baf676f6

creation_time 2023-01-09T14:36:04.149468544Z

custom_metadata map[]

id 9f1bfdf3-a461-2933-2a36-b136815bd32a

last_update_time 2023-01-09T14:36:04.149468544Z

local false

merged_from_canonical_ids <nil>

metadata <nil>

mount_accessor auth_userpass_8fabd83b

mount_path auth/userpass/

mount_type userpass

name mid

namespace_id root

Создадим теперь group к которой будет присоединена policy и Member Entity IDs, команда выглядит вот так:

vault write identity/group name="admins" type="internal" policies="admin" member_entity_ids=ID

1	vault write identity/group name="admins" type="internal" policies="admin" member_entity_ids=ID

вот результат:

[root@vault1 ~]# vault write identity/group name="admins" type="internal" policies="admin" member_entity_ids=98cb3fe3-6aac-8c02-7ea1-a576baf676f6
Key     Value
---     -----
id      35349401-d0fa-3451-d8a6-2d6a2cadb23e
name    admins

[root@vault1 ~]# vault write identity/group name="admins" type="internal" policies="admin" member_entity_ids=98cb3fe3-6aac-8c02-7ea1-a576baf676f6

Key Value

--- -----

id 35349401-d0fa-3451-d8a6-2d6a2cadb23e

name admins

Добавим нового пользователя в эту же группу admins:

[root@vault1 ~]# vault write auth/userpass/users/user2 password=123456789
Success! Data written to: auth/userpass/users/user2


[root@vault1 ~]# vault write identity/entity name=user2
Key        Value
---        -----
aliases    <nil>
id         d265af9e-6fc4-69f1-a2df-4da7fff75176
name       user2


[root@vault1 ~]# vault write identity/entity-alias name=user2 canonical_id=d265af9e-6fc4-69f1-a2df-4da7fff75176 mount_accessor=auth_userpass_8fabd83b
Key             Value
---             -----
canonical_id    d265af9e-6fc4-69f1-a2df-4da7fff75176
id              104d98d2-98b6-a8e1-e81d-5642fc76f570


[root@vault1 ~]# vault write identity/group name="admins" member_entity_ids=d265af9e-6fc4-69f1-a2df-4da7fff75176,98cb3fe3-6aac-8c02-7ea1-a576baf676f6
Success! Data written to: identity/group

[root@vault1 ~]# vault write auth/userpass/users/user2 password=123456789

Success! Data written to: auth/userpass/users/user2

[root@vault1 ~]# vault write identity/entity name=user2

Key Value

--- -----

aliases <nil>

id d265af9e-6fc4-69f1-a2df-4da7fff75176

name user2

[root@vault1 ~]# vault write identity/entity-alias name=user2 canonical_id=d265af9e-6fc4-69f1-a2df-4da7fff75176 mount_accessor=auth_userpass_8fabd83b

Key Value

--- -----

canonical_id d265af9e-6fc4-69f1-a2df-4da7fff75176

id 104d98d2-98b6-a8e1-e81d-5642fc76f570

[root@vault1 ~]# vault write identity/group name="admins" member_entity_ids=d265af9e-6fc4-69f1-a2df-4da7fff75176,98cb3fe3-6aac-8c02-7ea1-a576baf676f6

Success! Data written to: identity/group

обратите внимание что при добавлении к группе id entity нового пользователя нужно перечислять их все.

чтобы глянуть какие уже id есть в группе admins можем использовать команду:

vault read identity/group/name/admins | grep member_entity_ids

Аутентификация через gitlab

офф инструкция:

https://docs.gitlab.com/ee/integration/vault.html

идём в наш гитлаб:

добавляем:
http://localhost:8250/oidc/callback
http://127.0.0.1:8200/ui/vault/auth/oidc/oidc/callback

Application ID a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259

Secret 69fa2b783fc889b77cdfdfe20de999585d68183bcdd8341692b421e0dbacc598

переходим в консоль, логинимся в vault

[root@vault3 ~]# vault login
Token (will be hidden):

токен копируем отсюда:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.CAESIDXIPcxDW1Q8tZ66INGkA_x1SRW5JNjO481O3_4pzmVwGh4KHGh2cy43SHlMcmFVZGZ2Ynl5YmZKODhRWjJWU0I
token_accessor         GgACUsd9gNHtHoXzLBdugnjm
token_duration         596h31m18s
token_renewable        true
token_policies         ["default"]
identity_policies      ["admin-acces"]
policies               ["default" "admin-acces"]
token_meta_username    mid2

Success! You are now authenticated. The token information displayed below

is already stored in the token helper. You do NOT need to run "vault login"

again. Future Vault requests will automatically use this token.

Key Value

--- -----

token hvs.CAESIDXIPcxDW1Q8tZ66INGkA_x1SRW5JNjO481O3_4pzmVwGh4KHGh2cy43SHlMcmFVZGZ2Ynl5YmZKODhRWjJWU0I

token_accessor GgACUsd9gNHtHoXzLBdugnjm

token_duration 596h31m18s

token_renewable true

token_policies ["default"]

identity_policies ["admin-acces"]

policies ["default" "admin-acces"]

token_meta_username mid2

vault auth enable oidc

[root@vault3 ~]# vault auth enable oidc
Success! Enabled oidc auth method at: oidc/

1 2	[root@vault3 ~]# vault auth enable oidc Success! Enabled oidc auth method at: oidc/

теперь нужно передать application ID и secret сгенерированный в GitLab

пример:

vault write auth/oidc/config \
    oidc_discovery_url="https://gitlab.com" \
    oidc_client_id="your_application_id" \
    oidc_client_secret="your_secret" \
    default_role="demo" \
    bound_issuer="localhost"

vault write auth/oidc/config \

oidc_discovery_url="https://gitlab.com" \

oidc_client_id="your_application_id" \

oidc_client_secret="your_secret" \

default_role="demo" \

bound_issuer="localhost"

а вот то что я вставляю:

vault write auth/oidc/config \
    oidc_discovery_url="http://gitlab.local" \
    oidc_client_id="a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259" \
    oidc_client_secret="69fa2b783fc889b77cdfdfe20de999585d68183bcdd8341692b421e0dbacc598" \
    default_role="demo" \
    bound_issuer="localhost"

vault write auth/oidc/config \

oidc_discovery_url="http://gitlab.local" \

oidc_client_id="a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259" \

oidc_client_secret="69fa2b783fc889b77cdfdfe20de999585d68183bcdd8341692b421e0dbacc598" \

default_role="demo" \

bound_issuer="localhost"

[root@vault3 ~]# vault write auth/oidc/config \
>     oidc_discovery_url="http://gitlab.local" \
>     oidc_client_id="a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259" \
>     oidc_client_secret="69fa2b783fc889b77cdfdfe20de999585d68183bcdd8341692b421e0dbacc598" \
>     default_role="demo" \
>     bound_issuer="localhost"
Success! Data written to: auth/oidc/config

[root@vault3 ~]# vault write auth/oidc/config \

> oidc_discovery_url="http://gitlab.local" \

> oidc_client_id="a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259" \

> oidc_client_secret="69fa2b783fc889b77cdfdfe20de999585d68183bcdd8341692b421e0dbacc598" \

> default_role="demo" \

> bound_issuer="localhost"

Success! Data written to: auth/oidc/config

теперь надо записать роль

vault write auth/oidc/role/demo -<<EOF
{
   "user_claim": "sub",
   "allowed_redirect_uris": "your_vault_instance_redirect_uris",
   "bound_audiences": "your_application_id",
   "oidc_scopes": "openid",
   "role_type": "oidc",
   "policies": "demo",
   "ttl": "1h",
   "bound_claims": { "groups": ["yourGroup/yourSubgrup"] }
}
EOF

vault write auth/oidc/role/demo -<<EOF

{

"user_claim": "sub",

"allowed_redirect_uris": "your_vault_instance_redirect_uris",

"bound_audiences": "your_application_id",

"oidc_scopes": "openid",

"role_type": "oidc",

"policies": "demo",

"ttl": "1h",

"bound_claims": { "groups": ["yourGroup/yourSubgrup"] }

}

EOF

а вот что я буду вставлять:

vault write auth/oidc/role/demo -<<EOF
{
   "user_claim": "sub",
   "allowed_redirect_uris": "http://vault.test.local:8200/ui/vault/auth/oidc/oidc/callback",
   "bound_audiences": "a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259",
   "oidc_scopes": "openid",
   "role_type": "oidc",
   "policies": "demo",
   "ttl": "1h",
   "bound_claims": { "groups": ["admin-acces"] }
}
EOF

vault write auth/oidc/role/demo -<<EOF

{

"user_claim": "sub",

"allowed_redirect_uris": "http://vault.test.local:8200/ui/vault/auth/oidc/oidc/callback",

"bound_audiences": "a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259",

"oidc_scopes": "openid",

"role_type": "oidc",

"policies": "demo",

"ttl": "1h",

"bound_claims": { "groups": ["admin-acces"] }

}

EOF

результат такой:

[root@vault3 ~]# vault write auth/oidc/role/demo -<<EOF
> {
>    "user_claim": "sub",
>    "allowed_redirect_uris": "http://vault.test.local:8250/oidc/callback",
>    "bound_audiences": "a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259",
>    "oidc_scopes": "openid",
>    "role_type": "oidc",
>    "policies": "demo",
>    "ttl": "1h",
>    "bound_claims": { "groups": ["admin-acces"] }
> }
> EOF
Success! Data written to: auth/oidc/role/demo

[root@vault3 ~]# vault write auth/oidc/role/demo -<<EOF

> {

> "user_claim": "sub",

> "allowed_redirect_uris": "http://vault.test.local:8250/oidc/callback",

> "bound_audiences": "a4b9c2ce0b1a62d4e448450bdb4e458a123a6806549bb660e8179bf70ac13259",

> "oidc_scopes": "openid",

> "role_type": "oidc",

> "policies": "demo",

> "ttl": "1h",

> "bound_claims": { "groups": ["admin-acces"] }

> }

> EOF

Success! Data written to: auth/oidc/role/demo

всё можно заходить через гитлаб

6. Auto unseal hashicorp vault

идея в следующем, у нас есть 2 кластера hashicorp vault которые будут делать auto unseal друг друга. один кластер у нас на отдельных тачках а второй будет внутри k8s. Для простоты примера поставим minikub

6.1 установка minikub

отключаем selinux

[root@minikub ~]# cat /etc/selinux/config | grep -v ^#
SELINUX=disabled

ребутаем сервер.

ставим докер:

vi /etc/yum.repos.d/docker.repo

[docker]
baseurl=https://download.docker.com/linux/centos/7/x86_64/stable/
gpgcheck=0

[docker]

baseurl=https://download.docker.com/linux/centos/7/x86_64/stable/

gpgcheck=0

yum -y install docker-ce

systemctl start docker

systemctl enable docker

ставим Conntrack - он является частью платформы Netlifier. требуется для работы сложной сети Kubernetes, поскольку узлы должны отслеживать соединения между тысячами модулей и сервисов.

yum -y install conntrack

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

1	curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

chmod +x kubectl

mv kubectl /usr/local/bin/

wget https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

1	wget https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

chmod +x minikube-linux-amd64

mv minikube-linux-amd64 /usr/local/bin/minikube

minikube start

если при запуске мы получаем следующую ошибку:

[root@minikub ~]# minikube start
* minikube v1.28.0 on Centos 7.9.2009
* Automatically selected the docker driver. Other choices: none, ssh
* The "docker" driver should not be used with root privileges. If you wish to continue as root, use --force.
* If you are running minikube within a VM, consider using --driver=none:
*   https://minikube.sigs.k8s.io/docs/reference/drivers/none/

X Exiting due to DRV_AS_ROOT: The "docker" driver should not be used with root privileges.

[root@minikub ~]# minikube start

* minikube v1.28.0 on Centos 7.9.2009

* Automatically selected the docker driver. Other choices: none, ssh

* The "docker" driver should not be used with root privileges. If you wish to continue as root, use --force.

* If you are running minikube within a VM, consider using --driver=none:

* https://minikube.sigs.k8s.io/docs/reference/drivers/none/

X Exiting due to DRV_AS_ROOT: The "docker" driver should not be used with root privileges.

можно запустить с флагом --force - если работаете из под root
minikube start --force

если хотите работать из под пользователя то нужно добавить прав:

[root@minikub ~]# adduser mid
[root@minikub ~]# usermod -aG docker mid
[root@minikub ~]# su - mid
[mid@minikub ~]$ minikube start

включаем ingress для minikube

[root@minikub ~]# minikube addons enable ingress

и включим автодополнение для kubectl

[root@minikub ~]# yum install bash-completion -y

[root@minikub ~]# echo 'source <(kubectl completion bash)' >>~/.bashrc

6.2 подготовка хранилища для PV (nfs)

в качестве хранилища pv будем использовать nfs сервер.

192.168.1.176

настраиваем его:

yum install nfs-utils -y

systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap
systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

[root@nfs ~]# cat /etc/exports
/nfs 192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)

1 2	[root@nfs ~]# cat /etc/exports /nfs 192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)

создадим раздел (можно и не создавать а выбрать удобную для вас директорию, я захотел сделать отдельный раздел)

lvcreate -L 5G -n nfs centos
mkfs.ext4 -L nfs /dev/centos/nfs
mkdir /nfs
mount /dev/centos/nfs /nfs/
vi /etc/fstab

/dev/mapper/centos-nfs  /nfs                     ext4    defaults        1 2

1 2	/dev/mapper/centos-nfs /nfs ext4 defaults 1 2

добавляем права на запись:

[root@nfs ~]# chmod 777 /nfs

на клиент (minikub) тоже ставим nfs-client

yum install nfs-utils -y

теперь настраиваем провижинер (provisioner)

cat rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

apiVersion: v1

kind: ServiceAccount

metadata:

# replace with namespace where provisioner is deployed

namespace: default

---

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

rules:

- apiGroups: [""]

resources: ["nodes"]

verbs: ["get", "list", "watch"]

- apiGroups: [""]

resources: ["persistentvolumes"]

verbs: ["get", "list", "watch", "create", "delete"]

- apiGroups: [""]

resources: ["persistentvolumeclaims"]

verbs: ["get", "list", "watch", "update"]

- apiGroups: ["storage.k8s.io"]

resources: ["storageclasses"]

verbs: ["get", "list", "watch"]

- apiGroups: [""]

resources: ["events"]

verbs: ["create", "update", "patch"]

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

subjects:

- kind: ServiceAccount

# replace with namespace where provisioner is deployed

namespace: default

roleRef:

kind: ClusterRole

apiGroup: rbac.authorization.k8s.io

---

kind: Role

apiVersion: rbac.authorization.k8s.io/v1

metadata:

# replace with namespace where provisioner is deployed

namespace: default

rules:

- apiGroups: [""]

resources: ["endpoints"]

verbs: ["get", "list", "watch", "create", "update", "patch"]

---

kind: RoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

# replace with namespace where provisioner is deployed

namespace: default

subjects:

- kind: ServiceAccount

# replace with namespace where provisioner is deployed

namespace: default

roleRef:

kind: Role

apiGroup: rbac.authorization.k8s.io

cat nfs_class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'

parameters:

archiveOnDelete: "false"

cat nfs_provision.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.1.176
            - name: NFS_PATH
              value: /nfs
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.176
            path: /nfs

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: nfs-client-provisioner

# replace with namespace where provisioner is deployed

namespace: default

spec:

replicas: 1

strategy:

type: Recreate

selector:

matchLabels:

app: nfs-client-provisioner

template:

metadata:

labels:

app: nfs-client-provisioner

spec:

serviceAccountName: nfs-client-provisioner

containers:

- name: nfs-client-provisioner

image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2

volumeMounts:

- name: nfs-client-root

mountPath: /persistentvolumes

env:

- name: PROVISIONER_NAME

value: k8s-sigs.io/nfs-subdir-external-provisioner

- name: NFS_SERVER

value: 192.168.1.176

- name: NFS_PATH

value: /nfs

volumes:

- name: nfs-client-root

nfs:

server: 192.168.1.176

path: /nfs

применяем:

kubectl apply -f rbac.yaml
kubectl apply -f nfs_class.yaml
kubectl apply -f nfs_provision.yaml

теперь можем проверить создаётся ли PV

cat test-claim.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  storageClassName: nfs-client
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Mi

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

spec:

storageClassName: nfs-client

accessModes:

- ReadWriteMany

resources:

requests:

storage: 50Mi

kubectl apply -f test-claim.yaml

проверяем:

[root@minikub ~]# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-claim   Bound    pvc-1307b530-13ae-4dea-b5c2-076b0de5eed0   50Mi       RWX            nfs-client     116s
[root@minikub ~]#
[root@minikub ~]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE
pvc-1307b530-13ae-4dea-b5c2-076b0de5eed0   50Mi       RWX            Delete           Bound    default/test-claim   nfs-client              119s

[root@minikub ~]# kubectl get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

test-claim Bound pvc-1307b530-13ae-4dea-b5c2-076b0de5eed0 50Mi RWX nfs-client 116s

[root@minikub ~]#

[root@minikub ~]# kubectl get pv

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE

pvc-1307b530-13ae-4dea-b5c2-076b0de5eed0 50Mi RWX Delete Bound default/test-claim nfs-client 119s

как видим всё ок.
проверяем на nfs шаре:

[root@nfs ~]# ls -lah /nfs/
total 28K
drwxrwxrwx   4 root root 4.0K Jan 11 17:09 .
dr-xr-xr-x. 19 root root 4.0K Jan 11 13:02 ..
drwxrwxrwx   2 root root 4.0K Jan 11 17:09 default-test-claim-pvc-1307b530-13ae-4dea-b5c2-076b0de5eed0
drwx------   2 root root  16K Jan 11 13:02 lost+found
[root@nfs ~]#

[root@nfs ~]# ls -lah /nfs/

total 28K

drwxrwxrwx 4 root root 4.0K Jan 11 17:09 .

dr-xr-xr-x. 19 root root 4.0K Jan 11 13:02 ..

drwxrwxrwx 2 root root 4.0K Jan 11 17:09 default-test-claim-pvc-1307b530-13ae-4dea-b5c2-076b0de5eed0

drwx------ 2 root root 16K Jan 11 13:02 lost+found

[root@nfs ~]#

всё ок.

6.3 установка vault в minikub

ставить будем в namespace vault

[root@minikub ~]# kubectl create ns vault

будем использовать уже готовые helm чарты для vault

для начала установим helm:

https://github.com/helm/helm/releases/tag/v3.10.3

[root@minikub ~]# wget https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz

[root@minikub ~]# tar -zxvf helm-v3.10.3-linux-amd64.tar.gz

[root@minikub ~]# mv linux-amd64/helm /usr/local/bin/helm

Возьмём офф чарт:

https://github.com/hashicorp/vault-helm

[root@minikub ~]# git clone https://github.com/hashicorp/vault-helm.git

поправим values включим raft (можно оставить по умолчанию consul можно вообще в базу данных postgres всё отправить)

[root@minikub ~]# vim vault-helm/values-my.yaml

чтобы запустить на minikub нужно выключить affinity

injector:
  enabled: false

server:
  image:
    repository: "hashicorp/vault"
    tag: "1.8.4"
    pullPolicy: IfNotPresent
  logLevel: "info"
  logFormat: "json"
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m

  affinity: {}

  dataStorage:
    enabled: true
    size: 1Gi
    mountPath: "/vault/data"
    storageClass: nfs-client
    accessMode: ReadWriteOnce
  dev:
    enabled: false
  standalone:
    enabled: false
  ha:
    enabled: true
    replicas: 3

    # Set the api_addr configuration for Vault HA
    # See https://www.vaultproject.io/docs/configuration#api_addr
    # If set to null, this will be set to the Pod IP Address
    apiAddr: null


    raft:
      # Enables Raft integrated storage
      enabled: true


      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        storage "raft" {
          path = "/vault/data"
        }

        service_registration "kubernetes" {}

csi:
  enabled: true
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: 50m
      memory: 128Mi

injector:

enabled: false

server:

image:

repository: "hashicorp/vault"

tag: "1.8.4"

pullPolicy: IfNotPresent

logLevel: "info"

logFormat: "json"

resources:

requests:

memory: 256Mi

cpu: 250m

limits:

memory: 256Mi

cpu: 250m

affinity: {}

dataStorage:

enabled: true

size: 1Gi

mountPath: "/vault/data"

storageClass: nfs-client

accessMode: ReadWriteOnce

dev:

enabled: false

standalone:

enabled: false

ha:

enabled: true

replicas: 3

# Set the api_addr configuration for Vault HA

# See https://www.vaultproject.io/docs/configuration#api_addr

# If set to null, this will be set to the Pod IP Address

apiAddr: null

raft:

# Enables Raft integrated storage

enabled: true

config: |

ui = true

listener "tcp" {

tls_disable = 1

address = "[::]:8200"

cluster_address = "[::]:8201"

}

storage "raft" {

path = "/vault/data"

}

service_registration "kubernetes" {}

csi:

enabled: true

resources:

requests:

cpu: 50m

memory: 128Mi

limits:

cpu: 50m

memory: 128Mi

а вот такой вид будет при обычном запуске в кластере:

injector:
  enabled: false

server:
  image:
    repository: "hashicorp/vault"
    tag: "1.8.4"
    pullPolicy: IfNotPresent
  logLevel: "info"
  logFormat: "json"
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m

  dataStorage:
    enabled: true
    size: 1Gi
    mountPath: "/vault/data"
    storageClass: nfs-client
    accessMode: ReadWriteOnce
  dev:
    enabled: false
  standalone:
    enabled: false
  ha:
    enabled: true
    replicas: 3

    # Set the api_addr configuration for Vault HA
    # See https://www.vaultproject.io/docs/configuration#api_addr
    # If set to null, this will be set to the Pod IP Address
    apiAddr: null


    raft:
      # Enables Raft integrated storage
      enabled: true


      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        storage "raft" {
          path = "/vault/data"
        }

        service_registration "kubernetes" {}

csi:
  enabled: true
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: 50m
      memory: 128Mi

injector:

enabled: false

server:

image:

repository: "hashicorp/vault"

tag: "1.8.4"

pullPolicy: IfNotPresent

logLevel: "info"

logFormat: "json"

resources:

requests:

memory: 256Mi

cpu: 250m

limits:

memory: 256Mi

cpu: 250m

dataStorage:

enabled: true

size: 1Gi

mountPath: "/vault/data"

storageClass: nfs-client

accessMode: ReadWriteOnce

dev:

enabled: false

standalone:

enabled: false

ha:

enabled: true

replicas: 3

# Set the api_addr configuration for Vault HA

# See https://www.vaultproject.io/docs/configuration#api_addr

# If set to null, this will be set to the Pod IP Address

apiAddr: null

raft:

# Enables Raft integrated storage

enabled: true

config: |

ui = true

listener "tcp" {

tls_disable = 1

address = "[::]:8200"

cluster_address = "[::]:8201"

}

storage "raft" {

path = "/vault/data"

}

service_registration "kubernetes" {}

csi:

enabled: true

resources:

requests:

cpu: 50m

memory: 128Mi

limits:

cpu: 50m

memory: 128Mi

как видим разница только в
affinity: {}

ставим сам vault:
[root@minikub ~]# helm upgrade --install vault -n vault ./vault-helm/ -f ./vault-helm/values-my.yaml

теперь нам нужно поставить Secrets store CSI driver

он потребуется для подключения сикретов из vault в init контейнеры.

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

теперь нужно инициализировать vault

[root@minikub ~]# kubectl -n vault exec vault-0 -- vault operator init -key-shares=5 -key-threshold=3 -format=json > cluster-keys.json

1 2	[root@minikub ~]# kubectl -n vault exec vault-0 -- vault operator init -key-shares=5 -key-threshold=3 -format=json > cluster-keys.json

Получили ключ для unseal.

далее нам понадобится утилита jq поставим её:

yum install epel-release -y
yum install jq -y

cat cluster-keys.json | jq -r ".unseal_keys_b64[]"

[root@minikub ~]# cat cluster-keys.json | jq -r ".unseal_keys_b64[]"
sUwK9cVo+gZy2IDVryaHS0DHkGkXA9e1T0AQrzIRycHY
+CUfPu82zO/TD2osXmqiLIKliEKlbDV6SyT3ugshCK3k
XTRjmbHP/33VXy84eEXu/ioD5raBzIIImrzXz0XsBrnz
vv6sOVax0jBNQ0RRLJTbeYUVQOfDv5R/uJ/4B5+0oM2j
QrlfMN08AzRll1nsf27M6tDtT0iSHR0ZANAXMhio0968

[root@minikub ~]# cat cluster-keys.json | jq -r ".unseal_keys_b64[]"

sUwK9cVo+gZy2IDVryaHS0DHkGkXA9e1T0AQrzIRycHY

+CUfPu82zO/TD2osXmqiLIKliEKlbDV6SyT3ugshCK3k

XTRjmbHP/33VXy84eEXu/ioD5raBzIIImrzXz0XsBrnz

vv6sOVax0jBNQ0RRLJTbeYUVQOfDv5R/uJ/4B5+0oM2j

QrlfMN08AzRll1nsf27M6tDtT0iSHR0ZANAXMhio0968

далее распечатываем наше хранилище командой:

kubectl -n vault exec vault-0 -- vault operator unseal

ключи выше.
Если лень вводить руками то можно использовать цикл:

for i in `cat cluster-keys.json | jq -r ".unseal_keys_b64[]"`; do kubectl -n vault exec vault-0 -- vault operator unseal $i; done

1	for i in `cat cluster-keys.json \| jq -r ".unseal_keys_b64[]"`; do kubectl -n vault exec vault-0 -- vault operator unseal $i; done

результат будет такой:

[root@minikub ~]# for i in `cat cluster-keys.json | jq -r ".unseal_keys_b64[]"`; do kubectl -n vault exec vault-0 -- vault operator unseal $i; done
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       ec4e5f2b-dfac-762c-e33f-3c9e7817c1a5
Version            1.8.4
Storage Type       raft
HA Enabled         true
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       ec4e5f2b-dfac-762c-e33f-3c9e7817c1a5
Version            1.8.4
Storage Type       raft
HA Enabled         true
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.8.4
Storage Type            raft
Cluster Name            vault-cluster-d58a9688
Cluster ID              637ba7c8-8d57-5bb5-2840-c78cbaad841e
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.8.4
Storage Type            raft
Cluster Name            vault-cluster-d58a9688
Cluster ID              637ba7c8-8d57-5bb5-2840-c78cbaad841e
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.8.4
Storage Type            raft
Cluster Name            vault-cluster-d58a9688
Cluster ID              637ba7c8-8d57-5bb5-2840-c78cbaad841e
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24

[root@minikub ~]# for i in `cat cluster-keys.json | jq -r ".unseal_keys_b64[]"`; do kubectl -n vault exec vault-0 -- vault operator unseal $i; done

Key Value

--- -----

Seal Type shamir

Initialized true

Sealed true

Total Shares 5

Threshold 3

Unseal Progress 1/3

Unseal Nonce ec4e5f2b-dfac-762c-e33f-3c9e7817c1a5

Version 1.8.4

Storage Type raft

HA Enabled true