自动化部署 - 定制化RPM包构建

linux 上一些软件包的安装,往往需要很多手工介入的安装步骤,不利于进行大规模集群部署,而且免不了对一些软件需要定制化改造的需求,需要源码安装。如果需要一台台地去编译安装,那么运维就不用出机房了

文章以mysql打包构建为例,实现 基于源码构建定制化rpm包 和基于 二进制安装包构建rpm包 两种方式的的打包

打包好了,将该rpm包发布到构建的本地yum源,集群中所有机器一个yum install 命令就可以搞定了。如何在本地搭建yum源,可以见上一篇文章:自动化部署 - 构建YUM本地源

准备

安装需求包

1
yum -y install rpm-build redhat-rpm-config gcc gcc-c++ cmake make zlib-devel openssl-devel

创建rpmbuild构建目录:

1
2
mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
echo "%_topdir $HOME/rpmbuild" > ~/.rpmmacros

SPEC 文件分析

(1)文件头

一般的spec文件头包含以下几个域:

Summary:用一句话概括该软件包尽量多的信息。

Name:软件包的名字,最终RPM软件包是用该名字与版本号,释出号及体系号来命名软件包的。

Version:软件版本号。仅当软件包比以前有较大改变时才增加版本号。

Release: 软件包释出号。一般我们对该软件包做了一些小的补丁的时候就应该把释出号加1。

Vendor: 软件开发者的名字。

Copyright: 软件包所采用的版权规则。具体有:GPL(自由软件),BSD,MIT,Public Domain(公共域),Distributable(贡献),commercial(商业),Share(共享)等,一般的开发都写GPL。

Group:
软件包所属类别,具体类别有:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Amusements/Games (娱乐/游戏)
Amusements/Graphics(娱乐/图形)
Applications/Archiving (应用/文档)
Applications/Communications(应用/通讯)
Applications/Databases (应用/数据库)
Applications/Editors (应用/编辑器)
Applications/Emulators (应用/仿真器)
Applications/Engineering (应用/工程)
Applications/File (应用/文件)
Applications/Internet (应用/因特网)
Applications/Multimedia(应用/多媒体)
Applications/Productivity (应用/产品)
Applications/Publishing(应用/印刷)
Applications/System(应用/系统)
Applications/Text (应用/文本)
Development/Debuggers (开发/调试器)
Development/Languages (开发/语言)
Development/Libraries (开发/函数库)
Development/System (开发/系统)
Development/Tools (开发/工具)
Documentation (文档)
System Environment/Base(系统环境/基础)
System Environment/Daemons (系统环境/守护)
System Environment/Kernel (系统环境/内核)
System Environment/Libraries (系统环境/函数库)
System Environment/Shells (系统环境/接口)
User Interface/Desktops(用户界面/桌面)
User Interface/X (用户界面/X窗口)
User Interface/X Hardware Support (用户界面/X硬件支持)

Source:源程序软件包的名字。如 stardict-2.0.tar.gz。

%description:软件包详细说明,可写在多个行上。

2)%prep段

这个段是预处理段,通常用来执行一些解开源程序包的命令,为下一步的编译安装作准备。%prep和下面的%build,%install段一样,除了可以执行RPM所定义的宏命令(以%开头)以外,还可以执行SHELL命令,命令可以有很多行,如我们常写的tar解包命令。

(3)build段

本段是建立段,所要执行的命令为生成软件包服务,如make 命令。

(4)%install段

本段是安装段,其中的命令在安装软件包时将执行,如make install命令。

(5)%files段

本段是文件段,用于定义软件包所包含的文件,分为三类–说明文档(doc),配置文件(config)及执行程序,还可定义文件存取权限,拥有者及组别。

(6)%changelog段

本段是修改日志段。你可以将软件的每次修改记录到这里,保存到发布的软件包中,以便查询之用。每一个修改日志都有这样一种格式:第一行是:* 星期 月 日 年 修改人 电子信箱。其中:星期、月份均用英文形式的前3个字母,用中文会报错。接下来的行写的是修改了什么地方,可写多行。一般以减号开始,便于后续的查阅。

示例一:MYSQL 源码编译打包rpm

源码编译过程

源码–>预处理–>编译–>汇编–>链接–>执行

1
2
3
./configure
make
make install

源码编译打包rpm,所以在rpm 的 build段,需要完成生成mysql软件包服务,包含cmake和make操作。在install阶段,完成 make install操作。具体spec文件如下

cat mysql.spec

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
Name: ifengkou-mysql
Version:5.7.20
Release: 18
License: GPL
Group: applications/database
URL: https://www.percona.com/downloads/Percona-Server-5.7/Percona-Server-5.7.20-18/source/tarball/percona-server-5.7.20-18.tar.gz
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root
BuildRequires: cmake
Requires: coreutils,shadow-utils
Packager: ifengkou@hotmail.com
Autoreq: no
Source: percona-server-5.7.20-18.tar.gz
prefix: /opt/soft/ifengkou_mysql-%{version}-%{release}
Summary: percona server 5.7.20-18
%description
Percona Server is an enhanced, drop-in MySQL® replacement which has been downloaded more than 1,000,000 times.
A free open source solution, Percona Server is a MySQL alternative which offers breakthrough performance, scalability, features, and instrumentation. Self-tuning algorithms and support for extremely high-performance hardware make it the clear choice for organizations that demand excellent performance and reliability from their MySQL database server.
%define MYSQL_USER mysql
%define MYSQL_GROUP mysql
%define __os_install_post %{nil}
%prep
%setup -n %{name}-%{version}-%{release}
%build
#CFLAGS="-O3 -g -fno-exceptions -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing"
#CXX=g++
#CXXFLAGS="-O3 -g -fno-exceptions -fno-rtti -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing"
#export CFLAGS CXX CXXFLAGS
cmake . \
-DSYSCONFDIR:PATH=%{prefix} \
-DCMAKE_INSTALL_PREFIX:PATH=%{prefix} \
-DCMAKE_BUILD_TYPE:STRING=Release \
-DENABLE_PROFILING:BOOL=ON \
-DWITH_DEBUG:BOOL=OFF \
-DWITH_VALGRIND:BOOL=OFF \
-DENABLE_DEBUG_SYNC:BOOL=OFF \
-DWITH_EXTRA_CHARSETS:STRING=all \
-DWITH_SSL:STRING=bundled \
-DWITH_UNIT_TESTS:BOOL=OFF \
-DWITH_ZLIB:STRING=bundled \
-DWITH_PARTITION_STORAGE_ENGINE:BOOL=ON \
-DWITH_INNOBASE_STORAGE_ENGINE:BOOL=ON \
-DWITH_ARCHIVE_STORAGE_ENGINE:BOOL=ON \
-DWITH_BLACKHOLE_STORAGE_ENGINE:BOOL=ON \
-DWITH_PERFSCHEMA_STORAGE_ENGINE:BOOL=ON \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DENABLED_LOCAL_INFILE:BOOL=ON \
-DWITH_EMBEDDED_SERVER=0 \
-DINSTALL_LAYOUT:STRING=STANDALONE \
-DCOMMUNITY_BUILD:BOOL=ON \
-DWITHOUT_EXAMPLE_STORAGE_ENGINE=1 \
-DWITHOUT_NDBCLUSTER_STORAGE_ENGINE=1 \
-DENABLED_PROFILING=1 \
-DINNODB_PAGE_ATOMIC_REF_COUNT=1 \
-DWITH_INNODB_MEMCACHED=1 \
-DDOWNLOAD_BOOST=1 \
-DWITH_BOOST=/usr/local/boost_1_59_0;
make -j `cat /proc/cpuinfo | grep processor| wc -l`
%install
echo "-----------------"$RPM_BUILD_ROOT
make DESTDIR=$RPM_BUILD_ROOT install
cp %{_sourcedir}/my.cnf $RPM_BUILD_ROOT%{prefix}/
%clean
#rm -rf $RPM_BUILD_ROOT
rm -rf $RPM_BUILD_DIR/*
%files
%defattr(-, %{MYSQL_USER}, %{MYSQL_GROUP})
%attr(755, %{MYSQL_USER}, %{MYSQL_GROUP}) %{prefix}/*
%pre
if ! id %{MYSQL_USER} > /dev/null 2>&1;then
useradd -M -s /sbin/nologin %{MYSQL_USER}
fi
%post
if [ -f %{prefix}/support-files/mysql.server > /dev/null 2>&1 ] && [ ! -f %{_initddir}/mysql > /dev/null 2>&1 ];then
cp -f %{prefix}/support-files/mysql.server %{_initddir}/mysql
chmod +x %{_initddir}/mysql
chkconfig --level 2345 %{_initddir}/mysql on
fi
if [ ! -f %{_sysconfdir}/my.cnf ];then
cp %{prefix}/my.cnf %{_sysconfdir}/my.cnf
else
cp %{prefix}/my.cnf %{_sysconfdir}/my.cnf.rpmnew
fi
%preun
if [ -f %{_sysconfdir}/my.cnf ];then
mv %{_sysconfdir}/my.cnf %{_sysconfdir}/my.cnf.rpmold
fi
if [ -f %{_initddir}/mysql ];then
mv %{_initddir}/mysql %{_initddir}/mysql.rpmold
fi
%postun
rm -rf %{prefix}
userdel -r %{MYSQL_USER} >/dev/null 2>&1
%changelog

mysql源码包下载(下载地址),然后拷贝到~/rpmbuild/SOURCES目录,然后使用rpmbuild进行构建

1
2
3
cp percona-server-5.6.19-67.0.tar.gz ~/rpmbuild/SOURCES/
cp my.cnf ~/rpmbuild/SOURCES/ #拷贝模板配置文件
rpmbuild -bb mysql.spec

make 编译有点慢,需要几十分钟,还好这种软件只需要搞定一次,后续rpm 包安装的时候只需要make DESTDIR=$RPM_BUILD_ROOT install,拷贝文件到目标文件夹,这个还是非常快的。

  • rpmbuild -bb xxx.spec 只生成二进制格式的rpm包
  • rpmbuild -bs xxx.spec 只生成src格式的rpm包
  • rpmbuild -bp xxx.spec 只需要生成完整的源文件
  • rpmbuild -ba xxx.spec 完全打包,产生以上3个过程分别生成的包

安装过程中可能出现的一些异常

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
CMake Error at cmake/readline.cmake:66 (MESSAGE):
Curses library not found. Please install appropriate package,
remove CMakeCache.txt and rerun cmake.On Debian/Ubuntu, package name is libncurses5-dev, on Redhat and derivates it is ncurses-devel.
Call Stack (most recent call first):
cmake/readline.cmake:190 (FIND_CURSES)
cmake/readline.cmake:264 (FIND_SYSTEM_READLINE)
CMakeLists.txt:538 (MYSQL_CHECK_EDITLINE)
解决:yum install ncurses-devel
CMake Error at cmake/readline.cmake:266 (MESSAGE):
Cannot find system readline libraries.
Call Stack (most recent call first):
CMakeLists.txt:538 (MYSQL_CHECK_EDITLINE)
解决:yum install -y readline-devel
CMake Error at plugin/keyring_vault/CMakeLists.txt:18 (message):
Not building keyring_vault, could not find library: CURL
Call Stack (most recent call first):
plugin/keyring_vault/CMakeLists.txt:23 (CHECK_IF_LIB_FOUND)
解决:yum install libcurl-devel

如果make过程中,出现编译失败,很可能是内存不足导致,释放些内存就好了

最后生成的rpm包:

1
~/rpmbuild/RPMS/x86_64/ifengkou_mysql-5.7.20-18.x86_64.rpm

示例二:MYSQL 二进制包打包成rpm(推荐)

示例一因为有 make步骤,所以打包过程十分慢。定制化mysql,诉求一般是安装在指定位置,使用制定配置,安装卸载方便。所以并不需要从源码开始构建,直接从官方下载最新的二进制安装包

https://dev.mysql.com/downloads/mysql/5.7.html#downloads

下载 mysql-5.7.22-linux-glibc2.12-x86_64.tar.gz,拷贝到 ~/rpmbuild/SOURCES目录,我在这做了次重命名,所以需要先解压,重命名文件夹,再打包。

1
2
3
4
5
6
7
8
9
wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.22-linux-glibc2.12-x86_64.tar.gz
tar -xvzf mysql-5.7.22-linux-glibc2.12-x86_64.tar.gz
mv mysql-5.7.22-linux-glibc2.12-x86_64/* ifengkou-mysql-5.7.22/
tar czf ifengkou-mysql-5.7.22.tar.gz ifengkou-mysql-5.7.22
mv ifengkou-mysql-5.7.22.tar.gz ~/rpmbuild/SOURCES/

修改下spec 文件,cat mysql-5.7.22.spec

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
Summary : ifengkou mysql
Name : ifengkou-mysql
Version : 5.7.22
Release : 1
License : GPL
Source : ifengkou-mysql-5.7.22.tar.gz
BuildArch : x86_64
BuildRoot : root
AutoReqProv: no
AutoReq: no
Packager : ifengkou@hotmail.com
%define debug_package %{nil}
%description
This package is just a mysql RPM.
%prep
rm -rf /opt/soft/ifengkou-mysql
%setup -q
%build
%install
echo $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT/opt/soft/ifengkou-mysql
install -d -m 755 $RPM_BUILD_ROOT/opt/soft/ifengkou-mysql
cp -rf * $RPM_BUILD_ROOT/opt/soft/ifengkou-mysql
%post
rm -rf /etc/init.d/ifengkou-mysqld /etc/my.cnf
cp /opt/soft/ifengkou-mysql/support-files/mysql.server /etc/init.d/ifengkou-mysqld
cp /opt/soft/ifengkou-mysql/support-files/my.cnf /etc/my.cnf
rm -rf /etc/profile.d/ifengkou-mysql.sh
echo "export IFENGKOU_MYSQL_HOME=/opt/soft/ifengkou-mysql" >> /etc/profile.d/ifengkou-mysql.sh
echo "export PATH=\$IFENGKOU_MYSQL_HOME/bin:\$PATH" >> /etc/profile.d/ifengkou-mysql.sh
chmod +x /etc/init.d/ifengkou-mysqld
useradd mysql
chown -R mysql:mysql /opt/soft/ifengkou-mysql/data
mkdir /var/log/ifengkou-mysql
touch /var/log/ifengkou-mysql/mysql.log
chown -R mysql.mysql /var/log/ifengkou-mysql
mkdir /var/run/ifengkou-mysql
chown -R mysql.mysql /var/run/ifengkou-mysql
%check
%files
/opt/soft/ifengkou-mysql
%postun
rm -rf /opt/soft/ifengkou-mysql
rm -rf /etc/my.cnf
rm -rf /etc/profile.d/ifengkou-mysql.sh
rm -rf /var/log/ifengkou-mysql
rm -rf /var/run/ifengkou-mysql

最后打成了一个 rpmbuild/RPMS/x86_64/ifengkou-mysql-5.7.20-1.x86_64.rpm ,将rpm倒入到yum本地源后,直接
yum安装即可,这样在mysql 安装的时候就省去了很多手动配置参数的步骤,后续安装后启动
下面还包含了mysql-connector-java.jar 包的处理

1
2
3
4
5
6
7
8
9
10
11
12
13
echo "安装新的mysql..."
echo "yum -y install libaio libnuma* ifengkou-mysql -d 0 -e 0"
yum -y install libaio libnuma* ifengkou-mysql -d 0 -e 0
rsync -avvP ./tool/mysql-connector-java-5.1.37.jar /usr/share/java/
if [ -h "/usr/share/java/mysql-connector-java.jar" ];then
rm -f /usr/share/java/mysql-connector-java.jar
fi
ln -s /usr/share/java/mysql-connector-java-5.1.37.jar /usr/share/java/mysql-connector-java.jar
if [ $? = 0 ];then
echo "install mysql sucess"
else
echo "install mysql failed"
fi

安装后启动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
sed -i '/innodb_buffer_pool_size/c\innodb_buffer_pool_size=32G' /etc/my.cnf
sed -i '/max_allowed_packet/c\max_allowed_packet=25M' /etc/my.cnf
/sbin/chkconfig --add ifengkou-mysqld
systemctl enable ifengkou-mysqld.service
/etc/init.d/ifengkou-mysqld start
if [ $? = 0 ];then
echo "start mysql sucess"
else
echo "start mysql failed"
fi
. /etc/profile.d/ifengkou-mysql.sh

参考

Linux 源码编译安装MySQL 5.7
https://itbilu.com/database/mysql/VJVOut01M.html

打包mysql成rpm包
https://my.oschina.net/anthonyyau/blog/287063

RPM 打包技术与典型 SPEC 文件分析
https://www.ibm.com/developerworks/cn/linux/l-rpm/index.html

坚持原创技术分享,您的支持将鼓励我继续创作!
分享