ext fedora mbs

Fedora 模块性(Modularity)

简而言之,Modularity 可以扩展具有以下功能的 RPM 存储库:

  • 模块流(Module Stream)添加具有相同名称但从不同来源和不同构建配置构建的其他包。
  • 模块流可以有不同的生命周期。 例如,分发包中的包可以具有与分发包的生命周期不同的生命周期。
  • 模块流可以具有其组件的不同安装配置文件,即您可以定义一个或多个模块流配置文件,其中将包含应一起安装的 RPM 组。

Fedora 模块构建服务

Fedora 模块构建服务(Module Build Service, MBS)协调模块的构建过程,并负责以下任务: • 为客户端工具提供交互接口,以便提交模块和查询构建状态 • 验证输入的数据(比如modulemd、RPM SPEC等)是可用且正确的 • 在支持的构建系统(比如Koji)内准备构建环境 • 安排并构建模块组件,跟踪构建状态 • 在状态改变时发送总线(比如fedmsg、Apache MQ)信息,以便其他基础设施服务能够获取任务

Rocky Linux提供了一个不太完全的mbs安装指南:https://wiki.rockylinux.org/archive/legacy/mbs_installation/

本地构建模块

有关在线(貌似需要Fedora的有权限的ID)构建的教程,请参见:https://docs.fedoraproject.org/en-US/modularity/building-modules/fedora

先决条件

在开始本地构建模块之前,必须安装以下依赖项: $ sudo dnf install module-build-service fedpkg rpkg如果您没有打算以 root 身份运行构建命令,则必须将您的用户添加到 mock用户组: $ sudo usermod -a -G mock USERNAME $ newgrp从 distgit 构建你的模块 本地构建开始使用 fedpkg从你的 dist-git 仓库中。 例如,提交一个构建 testmodule:master模块,运行:

$ fedpkg clone modules/testmodule $ cd testmodule $ git checkout master $ fedpkg module-build-local

构建完成后,您将能够在~/modulebuild/builds/MODULE/results/ 找到构建结果。

另外,本地构建不支持流扩展。 如果您的模块依赖于另一个模块的多个流,例如 platform,您需要指定要构建的流。 例如: fedpkg module-build-local -s platform:f28.

部署 MBS

安装软件包

主要是安装 Apache 以及 MBS 的依赖。

dnf install fedmsg python3-gssapi git httpd mod_ssl python3-mod_wsgi python3-solv python3-pungi python3-psycopg2 mod_auth_gssapi module-build-service -y

配置并启动 Fedmsg

Fedmsg 是 Fedora 自己的一个消息服务,MBS 会使用 Fedmsg 在 Fedora 的各个基础设施之间相互协同。但我们不需要 Fedmsg 来协同,所以目前的配置只是运行 Fedmsg,但是并不向 Fedora 发送数据。因此需要改动一下默认配置,主要是注释掉 Fedora 的服务器与签名验证:

/etc/fedmsg.d/endpoints.py
------
    endpoints={
        # These are here so your local box can listen to the upstream
        # infrastructure's bus.  Cool, right?  :)
        "fedora-infrastructure": [
            # "tcp://hub.fedoraproject.org:9940", <- 注释此行
            # "tcp://stg.fedoraproject.org:9940",
        ],
        # "debian-infrastructure": [
        #    "tcp://fedmsg.olasd.eu:9940",
        # ],
/etc/fedmsg.d/module_build_service.py
------
config = {
    # Just for dev.
    "validate_signatures": False,  # <- 设置为 False
	  'sign_messages': False,   # <- 设置为 False
    # Talk to the relay, so things also make it to composer.stg in our dev env
    "active": True,
    # Since we're in active mode, we don't need to declare any of our own
    # passive endpoints.  This placeholder value needs to be here for the tests
    # to pass in Jenkins, though.  \o/
    "endpoints": {
        "fedora-infrastructure": [
            # Just listen to staging for now, not to production (spam!)
            # "tcp://hub.fedoraproject.org:9940", # <- 注释掉
            # "tcp://stg.fedoraproject.org:9940"
        ]
    },
    # Start of code signing configuration
    # 'sign_messages': True,  # <- 注释掉
    # 'validate_signatures': True,  # <- 注释掉
    # 'crypto_backend': 'x509',
    .....
}
/etc/fedmsg.d/ssl.py
------
config = dict(
    sign_messages=False,   # <- 设置为 False
    validate_signatures=False,   # <- 设置为 False
    ...)

这里的 Topic 也要改为自己的:

/etc/fedmsg.d/base.py
------
config = dict(
    # Prefix for the topic of each message sent.
    topic_prefix="org.fedora-plct", # <- 改为自己的名称

    # Set this to dev if you're hacking on fedmsg or an app.
    # Set to stg or prod if running in the Fedora Infrastructure
    environment="dev",
    ......)

用 Systemd 启动这两个服务:

systemctl enable fedmsg-hub --now
systemctl enable fedmsg-relay --now

mbs-frontend 的 Apache 配置

MBS 的前端是一个 wsgi 应用,不能直接运行,而应该配合 Apache Httpd 这样的服务器。新建以下配置文件:

/etc/httpd/conf.d/mbs.conf
------
<IfModule mod_ssl.c>
<VirtualHost *:443>
  ServerName mbs.gnulab.org
  WSGIDaemonProcess mbs user=mbs group=mbs threads=5
    WSGIScriptAlias / /etc/module-build-service/mbs.wsgi
  WSGIPassAuthorization on
    <Directory /etc/module-build-service>
        WSGIProcessGroup mbs
        WSGIApplicationGroup %{GLOBAL}
        Require all granted
    </Directory>
#    <Location />  # <- 测试时不使用 Krb 注释掉
#        AuthType GSSAPI
#        AuthName "GSSAPI Single Sign On Login"
#        GssapiCredStore keytab:/etc/koji.keytab
#        Require valid-user
#    </Location>

# 测试时不使用 SSL 注释掉
# SSLCertificateFile /etc/letsencrypt/live/mbs.gnulab.org/fullchain.pem 
# SSLCertificateKeyFile /etc/letsencrypt/live/mbs.gnulab.org/privkey.pem
# Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

也要新建 WGSI 文件:

/etc/module-build-service/mbs.wsgi
------
import logging
logging.basicConfig(level=logging.DEBUG)
from module_build_service import app as application

MBS 配置

接下来,我们为 MBS 新建用户并配置文件权限:

useradd mbs
passwd mbs
chown -R mbs:mbs /etc/module-build-service/
touch /tmp/module_build_service.log
chown mbs:fedmsg /tmp/module_build_service.log
chmod 664 /tmp/module_build_service.log

然后在 Koji 系统内添加 MBS 用户:

koji add-user mbs
koji call addBType module
koji grant-cg-access mbs module-build-service --new

MBS 有自己的 Koji 配置,按照实际情况来写:

/etc/module-build-service/koji.conf
------
[koji]

;url of XMLRPC server
server = https://openkoji.iscas.ac.cn/kojihub
;url of web interface
weburl = https://openkoji.iscas.ac.cn/koji
;url of package download site
topurl = https://openkoji.iscas.ac.cn/kojifiles
;path to the koji top directory
topdir = /mnt/koji

; configuration for SSL athentication
authtype = ssl

;client certificate
cert = /home/mbs/.koji/mbs.pem

;certificate of the CA that issued the HTTP server certificate
serverca = /home/mbs/.koji/koji_ca_cert.crt

最后是 MBS 本体的配置,按需修改:

/etc/module-build-service/config.py
------
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: MIT
from os import environ, path

# FIXME: workaround for this moment till confdir, dbdir (installdir etc.) are
# declared properly somewhere/somehow
confdir = path.abspath(path.dirname(__file__))
# use parent dir as dbdir else fallback to current dir
dbdir = path.abspath(path.join(confdir, "..")) if confdir.endswith("conf") else confdir


class ProdConfiguration(object):
    DEBUG = True
    # Make this random (used to generate session keys)
    SECRET_KEY = "74d9e9f9cd40e66fc6c4c2e9987dce48df3ce98542529126"
    SQLALCHEMY_DATABASE_URI = "sqlite:///{0}".format(path.join(dbdir, "module_build_service.db")) # 测试期间就用 SQLite 了
    #SQLALCHEMY_DATABASE_URI = 'postgresql://mbs:mysupersecretepasswordmbs@koji.gnulab.org/mbs'
    SQLALCHEMY_TRACK_MODIFICATIONS = True
    # Where we should run when running "manage.py run" directly.
    HOST = "0.0.0.0"
    PORT = 5000

    # Global network-related values, in seconds
    NET_TIMEOUT = 120
    NET_RETRY_INTERVAL = 30

    #DISTGITS = {"git+https://git.centos.org": ("git clone {repo_path}", "get_sources.sh")}
    SYSTEM = "koji"
    MESSAGING = "fedmsg"  # or amq
    MESSAGING_TOPIC_PREFIX = ["org.fedora-plct"]  # 修改为与 Fedmsg 配置一致
    KOJI_CONFIG = "/etc/module-build-service/koji.conf"
    KOJI_PROFILE = "koji"
    ARCHES = ["riscv64"]
    ALLOW_ARCH_OVERRIDE = False
    KOJI_REPOSITORY_URL = "https://openkoji.iscas.ac.cn/kojifiles"
    KOJI_TAG_PREFIXES = ["module", "scrmod"]
    KOJI_ENABLE_CONTENT_GENERATOR = True
    CHECK_FOR_EOL = False
    PDC_URL = "https://pdc.fedoraproject.org/rest_api/v1"
    PDC_INSECURE = False
    PDC_DEVELOP = True
    SCMURLS = ["https://src.fedoraproject.org"]
    YAML_SUBMIT_ALLOWED = True

    # How often should we resort to polling, in seconds
    # Set to zero to disable polling
    POLLING_INTERVAL = 600

    # Determines how many builds that can be submitted to the builder
    # and be in the build state at a time. Set this to 0 for no restrictions
    NUM_CONCURRENT_BUILDS = 5

    ALLOW_CUSTOM_SCMURLS = False

    RPMS_DEFAULT_REPOSITORY = " git+https://src.fedoraproject.org/rpms/"
    RPMS_ALLOW_REPOSITORY = True
    #RPMS_DEFAULT_CACHE = "http://pkgs.fedoraproject.org/repo/pkgs/"
    RPMS_ALLOW_CACHE = False

    MODULES_DEFAULT_REPOSITORY = "git+https://src.fedoraproject.org/modules/"
    MODULES_ALLOW_REPOSITORY = False
    MODULES_ALLOW_SCRATCH = True
    ALLOW_ONLY_COMPATIBLE_BASE_MODULES = True

    ALLOWED_GROUPS_TO_IMPORT_MODULE = set()

    # Available backends are: console and file
    LOG_BACKEND = "file"

    # Path to log file when LOG_BACKEND is set to "file".
    LOG_FILE = "/tmp/module_build_service.log"

    # Available log levels are: debug, info, warn, error.
    LOG_LEVEL = "debug"

    # Allow stream override
    ALLOW_STREAM_OVERRIDE_FROM_SCM = True

    # Settings for Kerberos
    KRB_KEYTAB = "/etc/mbs.keytab"
    KRB_PRINCIPAL = "mbs@GNULAB.ORG"

    # AMQ prefixed variables are required only while using 'amq' as messaging backend
    # Addresses to listen to
    AMQ_RECV_ADDRESSES = [
        "amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.koji",
        "amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.module_build_service",
    ]
    # Address for sending messages
    AMQ_DEST_ADDRESS = \
        "amqps://messaging.mydomain.com/Consumer.m8y.VirtualTopic.eng.module_build_service"
    AMQ_CERT_FILE = "/etc/module_build_service/msg-m8y-client.crt"
    AMQ_PRIVATE_KEY_FILE = "/etc/module_build_service/msg-m8y-client.key"
    AMQ_TRUSTED_CERT_FILE = "/etc/module_build_service/Root-CA.crt"

    # Disable Client Authorization
    NO_AUTH = True  # 测试或者内部使用可以关闭认证
    AUTH_METHOD = "kerberos"
    LDAP_URI = "ldap://koji.gnulab.org"
    LDAP_GROUPS_DN = "ou=group,dc=gnulab,dc=org"
    ADMIN_GROUPS = {"packageradmin"}
    ALLOWED_GROUPS = {"packager"}
    KOJI_CG_DEVEL_MODULE = True
    KOJI_PROXYUSER = True
    REBUILD_STRATEGY = 'only-changed'
    REBUILD_STRATEGY_ALLOW_OVERRIDE = True
    KOJI_CG_BUILD_TAG_TEMPLATE = "{}-modular-updates-candidate"
    KOJI_CG_DEFAULT_BUILD_TAG = "modular-updates-candidate"
    # Extra options set for newly created Koji tags
    KOJI_TAG_EXTRA_OPTS = {
        "mock.package_manager": "dnf",
        # This is needed to include all the Koji builds (and therefore
        # all the packages) from all inherited tags into this tag.
        # See https://pagure.io/koji/issue/588 and
        # https://pagure.io/fm-orchestrator/issue/660 for background.
        "repo_include_all": True,
        # Has been requested by Fedora infra in
        # https://pagure.io/fedora-infrastructure/issue/7620.
        # Disables systemd-nspawn for chroot.
        "mock.new_chroot": 0,
        # Works around fail-safe mechanism added in DNF 4.2.7
        # https://pagure.io/fedora-infrastructure/issue/8410
        "mock.yum.module_hotfixes": 1,
    }
    # DEFAULT_DIST_TAG_PREFIX = 'module_'
systemctl restart httpd
su mbs # 切换用户,防止文件权限问题
mbs-manager db upgrade

测试和使用 MBS

在使用 MBS 之前,需要导入平台模块。

su mbs

这是一个平台模块的示例

/home/mbs/platform-riscv-test.yaml 
------
document: modulemd
version: 1
data:
    name: platform
    stream: f37.0.0
    version: 3
    context: 00000000
    summary: Testing Rust dev base
    description: Testing Rust dev base
    license:
        module: [MIT]
    profiles:
        buildroot:
            rpms: [bash, bzip2, coreutils, cpio, diffutils, findutils, gawk, gcc, gcc-c++, grep, gzip, info, make, module-build-macros, patch,
redhat-rpm-config, rpm-build, sed, shadow-utils, tar, unzip, util-linux, which, xz]
        srpm-buildroot:
            rpms: [bash, gnupg2, module-build-macros, redhat-rpm-config, rpm-build, shadow-utils]
    xmd:
        mbs:
            buildrequires: {}
            commit: virtual
            requires: {}
            koji_tag: f37-build-side-32-misc-devel
            mse: TRUE
            virtual_streams: [f37]

然后我们用命令行管理工具导入:

mbs-manager import_module /home/mbs/platform-riscv-test.yaml 

模块的定义是一个 YAML 文件,下面就是一个示例:

---
document: modulemd
version: 2
data:
  name: zlib
  stream: '1.2.12'
  summary: zlib
  description: >
      zlib.      
  license:
    module:
      - MIT
  dependencies:
  - buildrequires:
      platform: []
    requires:
      platform: []
  components:
    rpms:
      zlib:
        rationale: zlib
        ref: f37
        buildorder: 0
...

使用 Wegt 发送该 YAML 到 MBS 服务器上面。

wget --quiet \
  --method POST \
  --header 'Content-Type: multipart/form-data; boundary=---011000010111000001101001' \
  --body-data '-----011000010111000001101001\r\nContent-Disposition: form-data; name="modulemd"\r\n\r\n---\ndocument: modulemd\nversion: 2\ndata:\n  name: zlib\n  stream: '\''1.2.12'\''\n  summary: zlib\n  description: >\n      zlib.\n  license:\n    module:\n      - MIT\n  dependencies:\n  - buildrequires:\n      platform: []\n    requires:\n      platform: []\n  components:\n    rpms:\n      zlib:\n        rationale: zlib\n        ref: f37\n        buildorder: 0\n...\r\n-----011000010111000001101001--\r\n' \
  --output-document \
  - 'http://127.0.0.1:18080/module-build-service/1/module-builds/?='

Last modified on 2024-09-10