Support for AARCH64

Tue Jun 16 15:38:48 UTC 2020

Ola,

Pål just pushed Monday's batch, so you get amd64 and aarch64 packages for
all the platforms. Go forth and test, the paint is still very wet.

Bonne journée!

-- 
Guillaume Quintard

On Tue, Jun 16, 2020 at 5:28 AM Emilio Fernandes <
emilio.fernandes70 at gmail.com> wrote:

> Hi,
>
> When we could expect the new aarch64 binaries at
> https://packagecloud.io/varnishcache/varnish-weekly ?
>
> Gracias!
> Emilio
>
> El mié., 15 abr. 2020 a las 14:33, Emilio Fernandes (<
> emilio.fernandes70 at gmail.com>) escribió:
>
>>
>>
>> El jue., 26 mar. 2020 a las 10:15, Martin Grigorov (<
>> martin.grigorov at gmail.com>) escribió:
>>
>>> Hello,
>>>
>>> Here is the PR: https://github.com/varnishcache/varnish-cache/pull/3263
>>> I will add some more documentation about the new setup.
>>> Any feedback is welcome!
>>>
>>
>> Nice work, Martin!
>>
>> Gracias!
>> Emilio
>>
>>
>>>
>>> Regards,
>>> Martin
>>>
>>> On Wed, Mar 25, 2020 at 9:55 PM Martin Grigorov <
>>> martin.grigorov at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Wed, Mar 25, 2020, 20:15 Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> is that script running as root?
>>>>>
>>>>
>>>> Yes.
>>>> I also added 'USER root' to its Dockerfile and '-u 0' to 'docker run'
>>>> arguments but it still doesn't work.
>>>> The x86 build is OK.
>>>> It must be something in the base docker image.
>>>> I've disabled the Alpine aarch64 job for now.
>>>> I'll send a PR tomorrow!
>>>>
>>>> Regards,
>>>> Martin
>>>>
>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>>
>>>>> On Wed, Mar 25, 2020 at 2:30 AM Martin Grigorov <
>>>>> martin.grigorov at gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I've moved 'dist' job to be executed in parallel with 'tar_pkg_tools'
>>>>>> and the results from both are shared in the workspace for the actual
>>>>>> packing jobs.
>>>>>> Now the new error for aarch64-apk job is:
>>>>>>
>>>>>> abuild: varnish >>> varnish: Updating the sha512sums in APKBUILD...
>>>>>> ]0; DEBUG: 4
>>>>>> ]0;abuild: varnish >>> varnish: Building /varnish 6.4.0-r1 (using
>>>>>> abuild 3.5.0-r0) started Wed, 25 Mar 2020 09:22:02 +0000
>>>>>> >>> varnish: Checking sanity of /package/APKBUILD...
>>>>>> >>> WARNING: varnish: No maintainer
>>>>>> >>> varnish: Analyzing dependencies...
>>>>>>   0%                                             %
>>>>>> ############################################>>> varnish: Installing for
>>>>>> build: build-base gcc libc-dev libgcc pcre-dev ncurses-dev libedit-dev
>>>>>> py-docutils linux-headers libunwind-dev python py3-sphinx
>>>>>> Waiting for repository lock
>>>>>> ERROR: Unable to lock database: Bad file descriptor
>>>>>> ERROR: Failed to open apk database: Bad file descriptor
>>>>>> >>> ERROR: varnish: builddeps failed
>>>>>> ]0; >>> varnish: Uninstalling dependencies...
>>>>>> Waiting for repository lock
>>>>>> ERROR: Unable to lock database: Bad file descriptor
>>>>>> ERROR: Failed to open apk database: Bad file descriptor
>>>>>>
>>>>>> Google suggested to do this:
>>>>>>    rm -rf /var/cache/apk
>>>>>>    mkdir /var/cache/apk
>>>>>>
>>>>>> It fails at 'abuild -r' -
>>>>>> https://github.com/martin-g/varnish-cache/blob/b62c357b389c0e1e31e9c001cbffb55090c2e49f/.circleci/make-apk-packages.sh#L61
>>>>>>
>>>>>> Any hints ?
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>> On Wed, Mar 25, 2020 at 2:39 AM Guillaume Quintard <
>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> So, you are pointing at the `dist` job, whose sole role is to
>>>>>>> provide us with a dist tarball, so we don't need that command line to work
>>>>>>> for everyone, just for that specific platform.
>>>>>>>
>>>>>>> On the other hand,
>>>>>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/.circleci/config.yml#L168 is
>>>>>>> closer to what you want, `distcheck` will be call on all platform, and you
>>>>>>> can see that it has the `--with-unwind` argument.
>>>>>>> --
>>>>>>> Guillaume Quintard
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 24, 2020 at 3:05 PM Martin Grigorov <
>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Mar 24, 2020, 17:19 Guillaume Quintard <
>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>
>>>>>>>>> Compare your configure line with what's currently in use (or the
>>>>>>>>> apkbuild file), there are a few options (with-unwind, without-jemalloc,
>>>>>>>>> etc.) That need to be set
>>>>>>>>>
>>>>>>>>
>>>>>>>> The configure line comes from "./autogen.des":
>>>>>>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/autogen.des#L35-L42
>>>>>>>> It is called at:
>>>>>>>>
>>>>>>>> https://github.com/varnishcache/varnish-cache/blob/4f9d8bed6b24bf9ee900c754f37615fdba1c44db/.circleci/config.yml#L40
>>>>>>>> In my branch at:
>>>>>>>>
>>>>>>>> https://github.com/martin-g/varnish-cache/blob/4b4626ee9cc366b032a45f27b54d77176125ef03/.circleci/make-apk-packages.sh#L26
>>>>>>>>
>>>>>>>> It fails only on aarch64 for Alpine Linux. The x86_64 build for
>>>>>>>> Alpine is fine.
>>>>>>>> AARCH64 for CentOS 7 and Ubuntu 18.04 are also fine.
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Mar 24, 2020, 08:05 Martin Grigorov <
>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 24, 2020 at 11:00 AM Martin Grigorov <
>>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 23, 2020 at 8:01 PM Guillaume Quintard <
>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for that.
>>>>>>>>>>>> A few remarks and questions:
>>>>>>>>>>>> - how much time does the "docker build" step takes? We can
>>>>>>>>>>>> possibly speed things up by push images to the dockerhub, as they don't
>>>>>>>>>>>> need to change very often.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Definitely such optimization would be a good thing to do!
>>>>>>>>>>> At the moment, with 'machine' executor it fetches the base image
>>>>>>>>>>> and then builds all the Docker layers again and again.
>>>>>>>>>>> Here are the timings:
>>>>>>>>>>> 1) Spinning up a VM - around 10secs
>>>>>>>>>>> 2) prepare env variables - 0secs
>>>>>>>>>>> 3) checkout code (varnish-cache) - 5secs
>>>>>>>>>>> 4) activate QEMU - 2secs
>>>>>>>>>>> 5) build packages
>>>>>>>>>>> 5.1) x86 deb - 3m 30secs
>>>>>>>>>>> 5.2) x86 rpm - 2m 50secs
>>>>>>>>>>> 5.3) aarch64 rpm - 35mins
>>>>>>>>>>> 5.4) aarch64 deb - 45mins
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> - any reason why you clone pkg-varnish-cache in each job? The
>>>>>>>>>>>> idea was to have it cloned once in tar-pkg-tools for consistency and
>>>>>>>>>>>> reproducibility, which we lose here.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I will extract the common steps once I see it working. This is
>>>>>>>>>>> my first CircleCI project and I still find my ways in it!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> - do we want to change things for the amd64 platforms for the
>>>>>>>>>>>> sake of consistency?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So far there is nothing specific for amd4 or aarch64, except the
>>>>>>>>>>> base Docker images.
>>>>>>>>>>> For example make-deb-packages.sh is reused for both amd64 and
>>>>>>>>>>> aarch64 builds. Same for -rpm- and now for -apk- (alpine).
>>>>>>>>>>>
>>>>>>>>>>> Once I feel the change is almost finished I will open a Pull
>>>>>>>>>>> Request for more comments!
>>>>>>>>>>>
>>>>>>>>>>> Martin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Mar 23, 2020 at 6:25 AM Martin Grigorov <
>>>>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 18, 2020 at 5:31 PM Martin Grigorov <
>>>>>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 4:35 PM Martin Grigorov <
>>>>>>>>>>>>>> martin.grigorov at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 3:23 PM Guillaume Quintard <
>>>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Offering arm64 packages requires a few things:
>>>>>>>>>>>>>>>> - arm64-compatible code (all good in
>>>>>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache)
>>>>>>>>>>>>>>>> - arm64-compatible package framework (all good in
>>>>>>>>>>>>>>>> https://github.com/varnishcache/pkg-varnish-cache)
>>>>>>>>>>>>>>>> - infrastructure to build the packages (uhoh, see below)
>>>>>>>>>>>>>>>> - infrastructure to store and deliver (
>>>>>>>>>>>>>>>> https://packagecloud.io/varnishcache)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, everything is in place, expect for the third point. At
>>>>>>>>>>>>>>>> the moment, there are two concurrent CI implementations:
>>>>>>>>>>>>>>>> - travis:
>>>>>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache/blob/master/.travis.yml It's
>>>>>>>>>>>>>>>> the historical one, and currently only runs compilation+test for OSX
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Actually it tests Linux AMD64 and ARM64 too.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - circleci:
>>>>>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache/blob/master/.circleci/config.yml the
>>>>>>>>>>>>>>>> new kid on the block, that builds all the packages and distchecks for all
>>>>>>>>>>>>>>>> the packaged platforms
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The issue is that cirecleci doesn't support arm64
>>>>>>>>>>>>>>>> containers (for now?), so we would need to re-implement the packaging logic
>>>>>>>>>>>>>>>> in Travis. It's not a big problem, but it's currently not a priority on my
>>>>>>>>>>>>>>>> side.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, I am totally ready to provide help if someone
>>>>>>>>>>>>>>>> wants to take that up. The added benefit it that Travis would be able to
>>>>>>>>>>>>>>>> handle everything and we can retire the circleci experiment
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I will take a look in the coming days and ask you if I need
>>>>>>>>>>>>>>> help!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've took a look at the current setup and here is what I've
>>>>>>>>>>>>>> found as problems and possible solutions:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) Circle CI
>>>>>>>>>>>>>> 1.1) problem - the 'machine' and 'Docker' executors run on
>>>>>>>>>>>>>> x86_64, so there is no way to build the packages in a "native" environment
>>>>>>>>>>>>>> 1.2) possible solutions
>>>>>>>>>>>>>> 1.2.1) use multiarch cross build
>>>>>>>>>>>>>> 1.2.2) use 'machine' executor that registers QEMU via
>>>>>>>>>>>>>> https://hub.docker.com/r/multiarch/qemu-user-static/ and
>>>>>>>>>>>>>> then builds and runs a custom Docker image that executes a shell script
>>>>>>>>>>>>>> with the build steps
>>>>>>>>>>>>>> It will look something like
>>>>>>>>>>>>>> https://github.com/yukimochi-containers/alpine-vpnserver/blob/69bb0a612c9df3e4ba78064d114751b760f0df9d/.circleci/config.yml#L19-L38 but
>>>>>>>>>>>>>> instead of uploading the Docker image as a last step it will run it.
>>>>>>>>>>>>>> The RPM and DEB build related code from current config.yml
>>>>>>>>>>>>>> will be extracted into shell scripts which will be copied in the custom
>>>>>>>>>>>>>> Docker images
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> From these two possible ways I have better picture in my head
>>>>>>>>>>>>>> how to do 1.2.2, but I don't mind going deep in 1.2.1 if this is what you'd
>>>>>>>>>>>>>> prefer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've decided to stay with Circle CI and use 'machine' executor
>>>>>>>>>>>>> with QEMU.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The changed config.yml could be seen at
>>>>>>>>>>>>> https://github.com/martin-g/varnish-cache/tree/feature/aarch64-packages/.circleci and
>>>>>>>>>>>>> the build at
>>>>>>>>>>>>> https://app.circleci.com/pipelines/github/martin-g/varnish-cache/71/workflows/3a275d79-62a9-48b4-9aef-1585de1c87c8
>>>>>>>>>>>>> The builds on x86 arch take 3-4 mins, but for aarch64
>>>>>>>>>>>>> (emulation!) ~40mins
>>>>>>>>>>>>> For now the jobs just build the .deb & .rpm packages for
>>>>>>>>>>>>> CentOS 7 and Ubuntu 18.04, both amd64 and aarch64.
>>>>>>>>>>>>> TODOs:
>>>>>>>>>>>>> - migrate Alpine
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> Build on Alpine aarch64 fails with:
>>>>>>>>>> ...
>>>>>>>>>> automake: this behaviour will change in future Automake versions:
>>>>>>>>>> they will
>>>>>>>>>> automake: unconditionally cause object files to be placed in the
>>>>>>>>>> same subdirectory
>>>>>>>>>> automake: of the corresponding sources.
>>>>>>>>>> automake: project, to avoid future incompatibilities.
>>>>>>>>>> parallel-tests: installing 'build-aux/test-driver'
>>>>>>>>>> lib/libvmod_debug/Makefile.am:12: warning:
>>>>>>>>>> libvmod_debug_la_LDFLAGS multiply defined in condition TRUE ...
>>>>>>>>>> lib/libvmod_debug/automake_boilerplate.am:19: ...
>>>>>>>>>> 'libvmod_debug_la_LDFLAGS' previously defined here
>>>>>>>>>> lib/libvmod_debug/Makefile.am:9:   'lib/libvmod_debug/
>>>>>>>>>> automake_boilerplate.am' included from here
>>>>>>>>>> + autoconf
>>>>>>>>>> + CONFIG_SHELL=/bin/sh
>>>>>>>>>> + export CONFIG_SHELL
>>>>>>>>>> + ./configure '--prefix=/opt/varnish' '--mandir=/opt/varnish/man'
>>>>>>>>>> --enable-maintainer-mode --enable-developer-warnings
>>>>>>>>>> --enable-debugging-symbols --enable-dependency-tracking
>>>>>>>>>> --with-persistent-storage --quiet
>>>>>>>>>> configure: WARNING: dot not found - build will fail if svg files
>>>>>>>>>> are out of date.
>>>>>>>>>> configure: WARNING: No system jemalloc found, using system malloc
>>>>>>>>>> configure: error: Could not find backtrace() support
>>>>>>>>>>
>>>>>>>>>> Does anyone know a workaround ?
>>>>>>>>>> I use multiarch/alpine:aarch64-edge as a base Docker image
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> - store the packages as CircleCI artifacts
>>>>>>>>>>>>> - anything else that is still missing
>>>>>>>>>>>>>
>>>>>>>>>>>>> Adding more architectures would be as easy as adding a new
>>>>>>>>>>>>> Dockerfile with a base image from the respective type.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) Travis CI
>>>>>>>>>>>>>> 2.1) problems
>>>>>>>>>>>>>> 2.1.1) generally Travis is slower than Circle!
>>>>>>>>>>>>>> Althought if we use CircleCI 'machine' executor it will be
>>>>>>>>>>>>>> slower than the current 'Docker' executor!
>>>>>>>>>>>>>> 2.1.2) Travis supports only Ubuntu
>>>>>>>>>>>>>> Current setup at CircleCI uses CentOS 7.
>>>>>>>>>>>>>> I guess the build steps won't have problems on Ubuntu.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3) GitHub Actions
>>>>>>>>>>>>>> GH Actions does not support ARM64 but it supports self hosted
>>>>>>>>>>>>>> ARM64 runners
>>>>>>>>>>>>>> 3.1) The problem is that there is no way to make a self
>>>>>>>>>>>>>> hosted runner really private. I.e. if someone forks Varnish Cache any
>>>>>>>>>>>>>> commit in the fork will trigger builds on the arm64 node. There is no way
>>>>>>>>>>>>>> to reserve the runner only for commits against
>>>>>>>>>>>>>> https://github.com/varnishcache/varnish-cache
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you see other problems or maybe different ways ?
>>>>>>>>>>>>>> Do you have preferences which way to go ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> varnish-dev mailing list
>>>>>>>>>>>>>>>> varnish-dev at varnish-cache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20200616/46f8f378/attachment-0001.html>