Arch Linux

kokizzu

Member

Registered: 2013-08-27

Posts: 19

Creating ScyllaDB Package

Hi, I've made https://github.com/scylladb/scylla package on AUR this morning, but after I upgrade my `pacman -Syu` and clear the compile cache, now no longer compilable. Any suggestion what's wrong and how to fix this? https://aur.archlinux.org/packages/scylla/
(warning: required 16GB RAM to compile if `-jn` where n is more than 1.

Current error: http://pastebin.com/pNMHwEfS

Previously built normally without error.

Today's upgrade:

apm-1.17.0-1  atom-1.15.0-1  binutils-2.28-1  bumblebee-3.2.1-15  chromium-57.0.2987.98-1  cifs-utils-6.7-1  deepin-icon-theme-15.12.34-1  dina-font-2.92-6
              flex-2.6.3-2  gcc-libs-multilib-6.3.1-2  gcc-multilib-6.3.1-2  geoip-database-20170307-1  glibc-2.25-1  gpgme-1.8.0-2  harfbuzz-1.4.5-1
              harfbuzz-icu-1.4.5-1  imagemagick-6.9.8.0-1  kaccounts-integration-16.12.3-1  kdialog-16.12.3-1  konsole-16.12.3-1  lib32-dbus-1.10.16-1
              lib32-e2fsprogs-1.43.4-1  lib32-gcc-libs-6.3.1-2  lib32-glibc-2.25-1  lib32-libglvnd-0.2.999+g4ba53457-1  lib32-libtasn1-4.10-1  lib32-mesa-17.0.1-2
              lib32-sqlite-3.17.0-1  lib32-wayland-1.13.0-1  libglvnd-0.2.999+g4ba53457-1  libinput-1.6.3-1  libkipi-16.12.3-1  libmatroska-1.4.7-1  libpurple-2.12.0-1
              libreoffice-still-5.2.6-1  libva-1.7.3-2  libxslt-1.1.29+41+gdf5330d1-1  linux-4.10.1-1  linux-api-headers-4.10.1-1  linux-firmware-20170227.5abb924-1
              linux-headers-4.10.1-1  linux-lts-4.9.14-1  linux-lts-headers-4.9.14-1  mesa-17.0.1-2  mobile-broadband-provider-info-20170310-1  nodejs-7.7.2-1
              noto-fonts-emoji-20170310-1  npm-4.4.1-1  openresolv-3.9.0-1  pidgin-2.12.0-1  primus-20151110-7  python-setuptools-1:34.3.2-1  python2-pillow-4.0.0-3
              python2-setuptools-1:34.3.2-1  refind-efi-0.10.5-1  signon-kwallet-extension-16.12.3-1  spectacle-16.12.3-1  tamsyn-font-1.11-2  thrift-0.10.0-1
              valgrind-3.12.0-2  virtualbox-5.1.16-1  virtualbox-ext-vnc-5.1.16-1  virtualbox-host-dkms-5.1.16-1  virtualgl-2.5.1-1  xf86-input-libinput-0.25.0-1

attica-qt5-5.32.0-1  breeze-icons-5.32.0-1  extra-cmake-modules-5.32.0-1  kactivities-5.32.0-1  karchive-5.32.0-1  kauth-5.32.0-1  kbookmarks-5.32.0-1
              kcmutils-5.32.0-1  kcodecs-5.32.0-1  kcompletion-5.32.0-1  kconfig-5.32.0-1  kconfigwidgets-5.32.0-1  kcoreaddons-5.32.0-1  kcrash-5.32.0-1
              kdbusaddons-5.32.0-1  kdeclarative-5.32.0-1  kded-5.32.0-1  kdelibs4support-5.32.0-1  kemoticons-5.32.0-1  kglobalaccel-5.32.0-1  kguiaddons-5.32.0-1
              ki18n-5.32.0-1  kiconthemes-5.32.0-1  kinit-5.32.0-1  kio-5.32.0-2  kitemmodels-5.32.0-1  kitemviews-5.32.0-1  kjobwidgets-5.32.0-1  knewstuff-5.32.0-1
              knotifications-5.32.0-1  knotifyconfig-5.32.0-1  kpackage-5.32.0-1  kparts-5.32.0-1  kpty-5.32.0-1  kservice-5.32.0-1  ktextwidgets-5.32.0-1
              kunitconversion-5.32.0-1  kwallet-5.32.0-1  kwayland-5.32.0-1  kwidgetsaddons-5.32.0-1  kwindowsystem-5.32.0-1  kxmlgui-5.32.0-1  oxygen-icons-1:5.32.0-1
              plasma-framework-5.32.0-1  solid-5.32.0-1  sonnet-5.32.0-1  thrift-0.10.0-1

Last `PKGBUILD`:

pkgname=scylla
pkg_major_minor=1.6
pkgver=${pkg_major_minor}.2
pkgrel=1
pkgdesc="NoSQL data store using the Seastar framework, compatible with Apache Cassandra"
arch=('x86_64')
url="http://www.scylladb.com/"
license=('AGPL')
depends=('antlr3' 'libantlr3c' 'libyaml' 'yaml-cpp' 'lz4' 'zlib' 'snappy'
		 'jsoncpp' 'gnutls' 'ninja' 'ragel' 'libaio' 'crypto++' 'xfsprogs' 'jre8-openjdk-headless'
		 'numactl' 'hwloc' 'libpciaccess' 'libxml2' 'python-pyparsing' 'boost-libs'
		 'lksctp-tools' 'protobuf' 'libunwind' 'systemtap-git' 'dpkg' 'xmlcutty-bin')
makedepends=('git' 'gcc')
provides=('scylla')
source=("git+https://github.com/scylladb/scylla.git#tag=scylla-${pkgver}"
'git+https://github.com/scylladb/scylla-ami.git'
'git+https://github.com/scylladb/scylla-swagger-ui.git'
'git+https://github.com/scylladb/scylla-seastar.git'
'git+https://github.com/scylladb/fmt.git'
'git+https://github.com/scylladb/dpdk.git'
'git+https://github.com/scylladb/c-ares.git'
)
md5sums=('SKIP'
'SKIP'
'SKIP'
'SKIP'
'SKIP'
'SKIP'
'SKIP'
)

# install dependencies
prepare() {

	# extract source zip
#	unzip -o scylla-${pkgver}.zip || echo 'ignore error'
#	cd ${pkgname}-${pkgname}-${pkgver}
	cd ${pkgname}
	#git checkout tags/${pkgname}-${pkgver}
	
	# init submodules
	git submodule init
#	git config submodule.scylla-seastar.url ${srcdir}/scylla-seastar
#	git config submodule.swagger-ui.url ${srcdir}/scylla-swagger-ui
#	git config submodule.dist/ami/files/scylla-ami.url ${srcdir}/scylla-ami
	git submodule update --init --recursive
	
	# init seastar
#	cp -R ${srcdir}/seastar/* seastar
#	cd seastar
#	git submodule init 
#	git config submodule.dpdk.url ${srcdir}/dpdk
#	git config submodule.fmt.url ${srcdir}/fmt
#	git config submodule.c-ares.url ${srcdir}/c-ares
#	git submodule update 
#	cd ..
	
	# downgrade thrift
	wget -c 'http://ala.seblu.net/packages/t/thrift/thrift-0.9.1-3-x86_64.pkg.tar.xz'
	sudo pacman --noconfirm -U thrift-0.9.1-3-x86_64.pkg.tar.xz

	# install antlr3++ headers
	DST=${srcdir}/antlr3
	if [ ! -d "$DST/.git" ]; then 
		rm -rf antlr3
		git clone --depth 1 https://github.com/antlr/antlr3.git
		sudo cp antlr3/runtime/Cpp/include/* /usr/include/
	fi
	
	# compile
	echo `pwd`
	./configure.py --mode=release --with=scylla --with=iotune --disable-xen --static --c-compiler=clang
	CORES=`lscpu | grep 0- | tail -n 1 | cut -d '-' -f 2`
	ninja build/release/scylla -j${CORES}
	ninja build/release/iotune -j${CORES}
	
	# download deb scripts
	URL=http://downloads.scylladb.com.s3.amazonaws.com/
	if [ ! -f pkgs.xml ]; then
		wget -c ${URL} -O pkgs.xml
	fi
	xmlcutty -path /ListBucketResult/Contents/Key -rename '\n' pkgs.xml | grep .deb | grep trusty_backup/scylladb-${pkg_major_minor}/ | grep -v '~rc' | grep -v '\-dbg' > pkgs.txt
	
	mkdir -p dpkg
	cat pkgs.txt | cut -d '/' -f 8 | cut -d '_' -f 1 | uniq > pkgs-unique.txt
	cat pkgs-unique.txt | while read line; do 
		FURL="$URL"`cat pkgs.txt | grep ${line} | tail -n 1`
		wget -c "$FURL"; 
		FNAME=`echo "$FURL" | cut -d '/' -f 11`
		echo "extracting $FNAME"
		dpkg-deb -x "$FNAME" dpkg/
	done
}

# packaging
package() {
#	DIR=${pkgname}-${pkgname}-${pkgver}
	DIR=${pkgname}
	src="$srcdir/$DIR/dpkg"

	# lib
	dst="$pkgdir/lib"
	mkdir -p ${dst}
	cp -Rv --preserve=timestamps "$src/lib/"* ${dst}

	# etc
	dst="$pkgdir/etc"
	mkdir -p ${dst}
	cp -Rv --preserve=timestamps "$src/etc/"* ${dst}
	
	src="$src/usr"
	
	# usr/share
	dst="$pkgdir/usr/share"
	mkdir -p ${dst}
	cp -Rv --preserve=timestamps "$src/share/"* ${dst}

	# usr/lib
	dst="$pkgdir/usr/lib"
	mkdir -p ${dst}
	cp -Rv --preserve=timestamps "$src/lib/"* ${dst}
		
	# remove missing symlink
	rm "$dst/scylla/jmx/symlinks/scylla-jmx"
	
	# usr/bin
	dst="$pkgdir/usr/bin"
	mkdir -p ${dst}
	cp -Rv --preserve=timestamps "$src/bin/"* ${dst}
	
	# overwrite with built 
	src="$srcdir/$DIR/build/release"
	install -p -m755 "$src/scylla" ${dst}
	install -p -m755 "$src/iotune" ${dst}
}

Lone_Wolf

Member

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 11,919

Re: Creating ScyllaDB Package

No idea what went wrong, but your PKGBUILD looks very weird.

The readme file section "Building and Running Scylla on Fedora" shows you only need 2 lines to build scylla from source.

./configure.py --mode=release --with=scylla --disable-xen
ninja-build build/release/scylla -j2 # you can use more cpus if you have tons of RAM

Edit :
are you trying to build  stable version 1.6.2 or a git version ?

Last edited by Lone_Wolf (2017-03-13 12:45:24)

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

(A works at time B)  && (time C > time B ) ≠  (A works at time C)

kokizzu

Member

Registered: 2013-08-27

Posts: 19

Re: Creating ScyllaDB Package

hi Lone_Wolf,

it's not that simple, because scylladb require old libraries and some others:
- thrift-0.9.1, current one on archlinux is 1.x
- antlr3++, does not exists
- scripts that only available on scylladb's rpm and deb

EDIT: i was trying both (zip) and (git, tags/scylla-1.6.2), i've tried with 1.6.1 and 1.6.2

Last edited by kokizzu (2017-03-14 04:33:29)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 11,553

Re: Creating ScyllaDB Package

This is completely the wrong way to go about it, though. wget'ing an old package and installing it with sudo inside the PKGBUILD? HELL NO!

Just a quick scan of the PKGBUILD and almost everything I see is wrong. I wouldn't even know where to start here.

Last edited by Scimmia (2017-03-14 04:37:54)

kokizzu

Member

Registered: 2013-08-27

Posts: 19

Re: Creating ScyllaDB Package

Hi, Scimmia, so what's the correct one?

Scimmia

Fellow

Registered: 2012-09-01

Posts: 11,553

Re: Creating ScyllaDB Package

Make a package that installs the old one in parallel with the current one.

You want to correct the bulk of the mistakes, keep in mind that nothing should be downloaded outside of the source array, and for the love of god, use a build function!