Support for AARCH64

Tue Jun 16 12:28:20 UTC 2020

Hi,

When we could expect the new aarch64 binaries at
https://packagecloud.io/varnishcache/varnish-weekly ?

Gracias!
Emilio

El mié., 15 abr. 2020 a las 14:33, Emilio Fernandes (<
emilio.fernandes70 at gmail.com>) escribió:

>
>
> El jue., 26 mar. 2020 a las 10:15, Martin Grigorov (<
> martin.grigorov at gmail.com>) escribió:
>
>> Hello,
>>
>> Here is the PR: https://github.com/varnishcache/varnish-cache/pull/3263
>> I will add some more documentation about the new setup.
>> Any feedback is welcome!
>>
>
> Nice work, Martin!
>
> Gracias!
> Emilio
>
>
>>
>> Regards,
>> Martin
>>
>> On Wed, Mar 25, 2020 at 9:55 PM Martin Grigorov <
>> martin.grigorov at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> On Wed, Mar 25, 2020, 20:15 Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> is that script running as root?
>>>>
>>>
>>> Yes.
>>> I also added 'USER root' to its Dockerfile and '-u 0' to 'docker run'
>>> arguments but it still doesn't work.
>>> The x86 build is OK.
>>> It must be something in the base docker image.
>>> I've disabled the Alpine aarch64 job for now.
>>> I'll send a PR tomorrow!
>>>
>>> Regards,
>>> Martin
>>>
>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>>
>>>> On Wed, Mar 25, 2020 at 2:30 AM Martin Grigorov <
>>>> martin.grigorov at gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I've moved 'dist' job to be executed in parallel with 'tar_pkg_tools'
>>>>> and the results from both are shared in the workspace for the actual
>>>>> packing jobs.
>>>>> Now the new error for aarch64-apk job is:
>>>>>
>>>>> abuild: varnish >>> varnish: Updating the sha512sums in APKBUILD...
>>>>> ]0; DEBUG: 4
>>>>> ]0;abuild: varnish >>> varnish: Building /varnish 6.4.0-r1 (using
>>>>> abuild 3.5.0-r0) started Wed, 25 Mar 2020 09:22:02 +0000
>>>>> >>> varnish: Checking sanity of /package/APKBUILD...
>>>>> >>> WARNING: varnish: No maintainer
>>>>> >>> varnish: Analyzing dependencies...
>>>>>   0%                                             %
>>>>> ############################################>>> varnish: Installing for
>>>>> build: build-base gcc libc-dev libgcc pcre-dev ncurses-dev libedit-dev
>>>>> py-docutils linux-headers libunwind-dev python py3-sphinx
>>>>> Waiting for repository lock
>>>>> ERROR: Unable to lock database: Bad file descriptor
>>>>> ERROR: Failed to open apk database: Bad file descriptor
>>>>> >>> ERROR: varnish: builddeps failed
>>>>> ]0; >>> varnish: Uninstalling dependencies...
>>>>> Waiting for repository lock
>>>>> ERROR: Unable to lock database: Bad file descriptor
>>>>> ERROR: Failed to open apk database: Bad file descriptor
>>>>>
>>>>> Google suggested to do this:
>>>>>    rm -rf /var/cache/apk
>>>>>    mkdir /var/cache/apk
>>>>>
>>>>> It fails at 'abuild -r' -
>>>>> https://github.com/martin-g/varnish-cache/blob/b62c357b389c0e1e31e9c001cbffb55090c2e49f/.circleci/make-apk-packages.sh#L61
>>>>>
>>>>> Any hints ?
>>>>>
>>>>> Martin
>>>>>
>>>>> On Wed, Mar 25, 2020 at 2:39 AM Guillaume Quintard <
>>>>> guillaume at varnish-software.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> So, you are pointing at the `dist` job, whose sole role is to provide
>>>>>> us with a dist tarball, so we don't need that command line to work for
>>>>>> everyone, just for that specific platform.
>>>>>>
>>>>>> On the other hand,
>>>>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/.circleci/config.yml#L168 is
>>>>>> closer to what you want, `distcheck` will be call on all platform, and you
>>>>>> can see that it has the `--with-unwind` argument.
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 24, 2020 at 3:05 PM Martin Grigorov <
>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 24, 2020, 17:19 Guillaume Quintard <
>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>
>>>>>>>> Compare your configure line with what's currently in use (or the
>>>>>>>> apkbuild file), there are a few options (with-unwind, without-jemalloc,
>>>>>>>> etc.) That need to be set
>>>>>>>>
>>>>>>>
>>>>>>> The configure line comes from "./autogen.des":
>>>>>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/autogen.des#L35-L42
>>>>>>> It is called at:
>>>>>>>
>>>>>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/.circleci/config.yml#L40
>>>>>>> In my branch at:
>>>>>>>
>>>>>>> https://github.com/martin-g/varnish-cache/blob/4b4626ee9cc366b032a45f27b54d77176125ef03/.circleci/make-apk-packages.sh#L26
>>>>>>>
>>>>>>> It fails only on aarch64 for Alpine Linux. The x86_64 build for
>>>>>>> Alpine is fine.
>>>>>>> AARCH64 for CentOS 7 and Ubuntu 18.04 are also fine.
>>>>>>>
>>>>>>> Martin
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Mar 24, 2020, 08:05 Martin Grigorov <
>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Tue, Mar 24, 2020 at 11:00 AM Martin Grigorov <
>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 23, 2020 at 8:01 PM Guillaume Quintard <
>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>
>>>>>>>>>>> Thank you for that.
>>>>>>>>>>> A few remarks and questions:
>>>>>>>>>>> - how much time does the "docker build" step takes? We can
>>>>>>>>>>> possibly speed things up by push images to the dockerhub, as they don't
>>>>>>>>>>> need to change very often.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Definitely such optimization would be a good thing to do!
>>>>>>>>>> At the moment, with 'machine' executor it fetches the base image
>>>>>>>>>> and then builds all the Docker layers again and again.
>>>>>>>>>> Here are the timings:
>>>>>>>>>> 1) Spinning up a VM - around 10secs
>>>>>>>>>> 2) prepare env variables - 0secs
>>>>>>>>>> 3) checkout code (varnish-cache) - 5secs
>>>>>>>>>> 4) activate QEMU - 2secs
>>>>>>>>>> 5) build packages
>>>>>>>>>> 5.1) x86 deb - 3m 30secs
>>>>>>>>>> 5.2) x86 rpm - 2m 50secs
>>>>>>>>>> 5.3) aarch64 rpm - 35mins
>>>>>>>>>> 5.4) aarch64 deb - 45mins
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> - any reason why you clone pkg-varnish-cache in each job? The
>>>>>>>>>>> idea was to have it cloned once in tar-pkg-tools for consistency and
>>>>>>>>>>> reproducibility, which we lose here.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I will extract the common steps once I see it working. This is my
>>>>>>>>>> first CircleCI project and I still find my ways in it!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> - do we want to change things for the amd64 platforms for the
>>>>>>>>>>> sake of consistency?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So far there is nothing specific for amd4 or aarch64, except the
>>>>>>>>>> base Docker images.
>>>>>>>>>> For example make-deb-packages.sh is reused for both amd64 and
>>>>>>>>>> aarch64 builds. Same for -rpm- and now for -apk- (alpine).
>>>>>>>>>>
>>>>>>>>>> Once I feel the change is almost finished I will open a Pull
>>>>>>>>>> Request for more comments!
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 23, 2020 at 6:25 AM Martin Grigorov <
>>>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 18, 2020 at 5:31 PM Martin Grigorov <
>>>>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 4:35 PM Martin Grigorov <
>>>>>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 3:23 PM Guillaume Quintard <
>>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Offering arm64 packages requires a few things:
>>>>>>>>>>>>>>> - arm64-compatible code (all good in
>>>>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache)
>>>>>>>>>>>>>>> - arm64-compatible package framework (all good in
>>>>>>>>>>>>>>> https://github.com/varnishcache/pkg-varnish-cache)
>>>>>>>>>>>>>>> - infrastructure to build the packages (uhoh, see below)
>>>>>>>>>>>>>>> - infrastructure to store and deliver (
>>>>>>>>>>>>>>> https://packagecloud.io/varnishcache)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, everything is in place, expect for the third point. At
>>>>>>>>>>>>>>> the moment, there are two concurrent CI implementations:
>>>>>>>>>>>>>>> - travis:
>>>>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache/blob/master/.travis.yml It's
>>>>>>>>>>>>>>> the historical one, and currently only runs compilation+test for OSX
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Actually it tests Linux AMD64 and ARM64 too.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - circleci:
>>>>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache/blob/master/.circleci/config.yml the
>>>>>>>>>>>>>>> new kid on the block, that builds all the packages and distchecks for all
>>>>>>>>>>>>>>> the packaged platforms
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The issue is that cirecleci doesn't support arm64 containers
>>>>>>>>>>>>>>> (for now?), so we would need to re-implement the packaging logic in Travis.
>>>>>>>>>>>>>>> It's not a big problem, but it's currently not a priority on my side.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, I am totally ready to provide help if someone wants
>>>>>>>>>>>>>>> to take that up. The added benefit it that Travis would be able to handle
>>>>>>>>>>>>>>> everything and we can retire the circleci experiment
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will take a look in the coming days and ask you if I need
>>>>>>>>>>>>>> help!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've took a look at the current setup and here is what I've
>>>>>>>>>>>>> found as problems and possible solutions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) Circle CI
>>>>>>>>>>>>> 1.1) problem - the 'machine' and 'Docker' executors run on
>>>>>>>>>>>>> x86_64, so there is no way to build the packages in a "native" environment
>>>>>>>>>>>>> 1.2) possible solutions
>>>>>>>>>>>>> 1.2.1) use multiarch cross build
>>>>>>>>>>>>> 1.2.2) use 'machine' executor that registers QEMU via
>>>>>>>>>>>>> https://hub.docker.com/r/multiarch/qemu-user-static/ and then
>>>>>>>>>>>>> builds and runs a custom Docker image that executes a shell script with the
>>>>>>>>>>>>> build steps
>>>>>>>>>>>>> It will look something like
>>>>>>>>>>>>> https://github.com/yukimochi-containers/alpine-vpnserver/blob/69bb0a612c9df3e4ba78064d114751b760f0df9d/.circleci/config.yml#L19-L38 but
>>>>>>>>>>>>> instead of uploading the Docker image as a last step it will run it.
>>>>>>>>>>>>> The RPM and DEB build related code from current config.yml
>>>>>>>>>>>>> will be extracted into shell scripts which will be copied in the custom
>>>>>>>>>>>>> Docker images
>>>>>>>>>>>>>
>>>>>>>>>>>>> From these two possible ways I have better picture in my head
>>>>>>>>>>>>> how to do 1.2.2, but I don't mind going deep in 1.2.1 if this is what you'd
>>>>>>>>>>>>> prefer.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I've decided to stay with Circle CI and use 'machine' executor
>>>>>>>>>>>> with QEMU.
>>>>>>>>>>>>
>>>>>>>>>>>> The changed config.yml could be seen at
>>>>>>>>>>>> https://github.com/martin-g/varnish-cache/tree/feature/aarch64-packages/.circleci and
>>>>>>>>>>>> the build at
>>>>>>>>>>>> https://app.circleci.com/pipelines/github/martin-g/varnish-cache/71/workflows/3a275d79-62a9-48b4-9aef-1585de1c87c8
>>>>>>>>>>>> The builds on x86 arch take 3-4 mins, but for aarch64
>>>>>>>>>>>> (emulation!) ~40mins
>>>>>>>>>>>> For now the jobs just build the .deb & .rpm packages for CentOS
>>>>>>>>>>>> 7 and Ubuntu 18.04, both amd64 and aarch64.
>>>>>>>>>>>> TODOs:
>>>>>>>>>>>> - migrate Alpine
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> Build on Alpine aarch64 fails with:
>>>>>>>>> ...
>>>>>>>>> automake: this behaviour will change in future Automake versions:
>>>>>>>>> they will
>>>>>>>>> automake: unconditionally cause object files to be placed in the
>>>>>>>>> same subdirectory
>>>>>>>>> automake: of the corresponding sources.
>>>>>>>>> automake: project, to avoid future incompatibilities.
>>>>>>>>> parallel-tests: installing 'build-aux/test-driver'
>>>>>>>>> lib/libvmod_debug/Makefile.am:12: warning:
>>>>>>>>> libvmod_debug_la_LDFLAGS multiply defined in condition TRUE ...
>>>>>>>>> lib/libvmod_debug/automake_boilerplate.am:19: ...
>>>>>>>>> 'libvmod_debug_la_LDFLAGS' previously defined here
>>>>>>>>> lib/libvmod_debug/Makefile.am:9:   'lib/libvmod_debug/
>>>>>>>>> automake_boilerplate.am' included from here
>>>>>>>>> + autoconf
>>>>>>>>> + CONFIG_SHELL=/bin/sh
>>>>>>>>> + export CONFIG_SHELL
>>>>>>>>> + ./configure '--prefix=/opt/varnish' '--mandir=/opt/varnish/man'
>>>>>>>>> --enable-maintainer-mode --enable-developer-warnings
>>>>>>>>> --enable-debugging-symbols --enable-dependency-tracking
>>>>>>>>> --with-persistent-storage --quiet
>>>>>>>>> configure: WARNING: dot not found - build will fail if svg files
>>>>>>>>> are out of date.
>>>>>>>>> configure: WARNING: No system jemalloc found, using system malloc
>>>>>>>>> configure: error: Could not find backtrace() support
>>>>>>>>>
>>>>>>>>> Does anyone know a workaround ?
>>>>>>>>> I use multiarch/alpine:aarch64-edge as a base Docker image
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> - store the packages as CircleCI artifacts
>>>>>>>>>>>> - anything else that is still missing
>>>>>>>>>>>>
>>>>>>>>>>>> Adding more architectures would be as easy as adding a new
>>>>>>>>>>>> Dockerfile with a base image from the respective type.
>>>>>>>>>>>>
>>>>>>>>>>>> Martin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Travis CI
>>>>>>>>>>>>> 2.1) problems
>>>>>>>>>>>>> 2.1.1) generally Travis is slower than Circle!
>>>>>>>>>>>>> Althought if we use CircleCI 'machine' executor it will be
>>>>>>>>>>>>> slower than the current 'Docker' executor!
>>>>>>>>>>>>> 2.1.2) Travis supports only Ubuntu
>>>>>>>>>>>>> Current setup at CircleCI uses CentOS 7.
>>>>>>>>>>>>> I guess the build steps won't have problems on Ubuntu.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3) GitHub Actions
>>>>>>>>>>>>> GH Actions does not support ARM64 but it supports self hosted
>>>>>>>>>>>>> ARM64 runners
>>>>>>>>>>>>> 3.1) The problem is that there is no way to make a self hosted
>>>>>>>>>>>>> runner really private. I.e. if someone forks Varnish Cache any commit in
>>>>>>>>>>>>> the fork will trigger builds on the arm64 node. There is no way to reserve
>>>>>>>>>>>>> the runner only for commits against
>>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you see other problems or maybe different ways ?
>>>>>>>>>>>>> Do you have preferences which way to go ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> varnish-dev mailing list
>>>>>>>>>>>>>>> varnish-dev at varnish-cache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20200616/09880404/attachment-0001.html>