From dridi at varni.sh Thu Oct 12 15:58:42 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 12 Oct 2017 17:58:42 +0200 Subject: VIP20: Varnish ABI and packaging Message-ID: Hello everyone, As requested last week, I started a VIP regarding VMODs, Varnish ABI, and packaging. While this is still incomplete, please let me know if anything needs a clarification so far. The research took me a while so I'll continue later, probably next week. https://github.com/varnishcache/varnish-cache/wiki/VIP20%3A-Varnish-ABI-and-packaging Cheers, Dridi From phk at phk.freebsd.dk Mon Oct 16 10:35:14 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 16 Oct 2017 10:35:14 +0000 Subject: draft VDD17 minutes Message-ID: <2216.1508150114@critter.freebsd.dk> I'm resending these, they're the subject of todays bugwash... Minutes of the 20171009 VDD =========================== This VDD was a small, by-invitation-only VDD to attempt to nail down the big changes in the 201803 release cycle. Participants: phk, slink, geoff, martin, daghf Location: @phk, Slagelse, Denmark Type specific methods --------------------- An attempt to enable the VCL syntax look more oo-ish/python-ish with a heuristic definition of "per-type-methods". Strawman: vmod1.vcc: $Function REAL foo(STRING, INT) vmod2.vcc: $Function BACKEND bar(REAL, STRING) Today these would combine as: bereq.backend = vmod2.bar(vmod1.foo("something", 12), "other"); With the new syntax it could also be written as: bereq.backend = vmod1.foo("something", 12)->vmod2.bar("other"); The '->' takes as left-hand side an expression, and as right hand side a function or procedure call where the first non-PRIV argument must match the type of the left-hand expression (subject to automatic foldings). We introduce the new '->' operator for this because using the '.' is at least confusing: bereq.backend = vmod1.foo("something", 12).vmod2.bar("other"); We talked about changing to ':' or '::' for name scoping instead, but decided it would break too much VCL: bereq.backend = vmod1::foo("something", 12).vmod2::bar("other"); This is VCL4.0 compatible, but should cause a minor bumb (VCL4.1) VMOD Function pointers ---------------------- Enable VMODs calling of functions determined at VCC-time. Strawman: vmod1.vcc: $Type BOOL match_func(STRINGS) $Function INT count_req_x(match_func) vmod2.vcc: $Function BOOL match_cookie(STRINGS) vcl: sub vcl_recv { if (vmod1.count_req_x(vmod2.match_cookie) > 3)) { return(error, 400); } } At this time VCL functions have no return types, but when they grow that at a future date, they could also be used. VMOD importing VMOD ------------------- Allow vmods to call other vmods Strawman: vmod1.vcc: $Object foo(STRING) $Method bar(int) vmod2.vcc: $Import vmod1 [from "..."] $Function INT fnord(vmod1.foo, STRING) vmod2.c: INT vmod_fnord(VRT_CTX, VMOD1_FOO foo, STRING str) { [...] VMOD1_FOO_BAR(foo, 12) [...] } This one has a number of tricky issues, for instance VCC needs to create wrapper functions when there are PRIV arguments etc. Current use-case examples are not particularly strong. vcl_init object consistency --------------------------- Today objects can be left uncreated in vcl_init{} with conditional constructs: sub vcl_init { if (something) { new foo = vmod.bar(); } } This should be legal, and all code should guard against NULL object pointers. However, leaving an object uninitialized this way, particularly in deep if-elif-elif... constructs is also a typical error, so we should by default complain about any objects not created at the end of vcl_init{} (probably controlled by the vcc_err_unref parameter) The check that all objects have been initialized would happen at the tail end of VGC_Load() (== VCL_EVENT_LOAD), and thus it will be vcl.load which spots the problem and fails. XXX: Can VCL cold->warm transition also fail due to changed conditional excution? XXX: Still need to fix _fini destruction order (#2297) In re #1799 ----------- Consider a VFP/VDP style list of "veto functions" called in sequence under lock from HSH_Lookup() BOOL veto(struct objcore *)) Default list "rfc2616 vary" (better ? "ttl vary") This will allow VMODS implementing things like cache-key: http://httpwg.org/http-extensions/key.html VMOD object references ---------------------- Vmods must not hold references, explicit or implicit, to foreign objects outside their activation. This means that arguments to VMOD functions in particular strings and backends only can be considered valid during the call of the function. 201803 release VCL ------------------ The goal is to support both VCL4.0 and new VCL version, selecting on the "vcl" declaration in the VCL. STRING_LIST implementation -------------------------- Change from varargs to struct string_list { int argc; const char * const *argv; } VCC knows how many elements will be required, and the struct will be allocated on the stack of the current context. Workspace max used VSL ---------------------- Record final WS usage and high-water mark in VSL, use varnishhist() to ponder. New WS_highwater() function needed to record size of reserve-free and maybe lost-to-snapshots high water. Extensible workspaces --------------------- New parameter, default off, enables extensible workspaces. When out of workspace, new slabs of WS are malloc'ed and links to the old replaced WS. On fini walk list and free(3). Size of extension segments need to be thought about/decided. VMODs should check for failure ------------------------------ Vmod functions should check early if VCL has already failed, and return cheap compatible return value without doing work. vrt.h should provide macro for this, something like: #define ALREADY_GOING_DOWN_IN_FLAMES(x) \ do { \ if(*ctx->handling) \ return(x); \ } while(0) VCL variables ------------- Strawman: var TYPE name; var TYPE .var.name; # req.var.foo, bereq.var.bar, ... Does _not_ allow for VMOD objects which are created with "new". Smaller scope VCL objects ------------------------- Allow "new ..." outside vcl_init{} This affects the arguments to the object creator functions, in particular there will be no CLI available to whine to. Drop PRIV_CALL -------------- PRIV_CALL was more or less entirely invented so that a regexp VMOD had some place to store the compiled regexp, at a time when we did not have VMOD objects. Only in-tree user is std.fileread() which caches the file opened. The tacit assumption is that the argument is constant so that the cached derivation of it can be reused, this prevents usage such as: std.fileread(req.url) With proper vmod object usage, and more dynamic object lifetimes this could be possible: vcl_init { new motd = std.fileread("/etc/motd"); } vcl_recv { set req.http.motd = motd.read(); set req.http.bad = std.fileread("/etc/passwd").read(); } The last line depends on allowing VMOD object creation outside of vcl_init{}. VMOD polymorphism ----------------- Make VCC responsible for picking best prototype: vmod1.vcc: $Function INT foo(STRING) $Function INT foo(BACKEND) $Function INT foo(STRING_LIST) XXX: Not sure if we should also allow return value to vary. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Oct 30 10:20:40 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 30 Oct 2017 10:20:40 +0000 Subject: Gimpel Softwares PC-lint Plus is a no-go. Message-ID: <64628.1509358840@critter.freebsd.dk> As some of you have heard, I have been beta-testing Gimpel Softwares new PC-lint Plus product. The good news is that it is a tool which shows a lot of promise from a very skilled organization. The bad news is that their new licensing is too expensive and legally unworkable for Open Source projects. Since I have evangelized enthusiastically for their previous product, this update on my position seems required. Pricing: -------- The single-user windows-license of the previous product "PCLint" cost $389 for a single seat, and $1000 for the obfuscated-source "FlexeLint" version which could run on UNIX systems. I like FlexeLint, and while I found the $600 "UNIX-tax" unreasonable and discriminatory, I paid for a license at the steeper price. And I've been praising and recommending the product to everybody who cared to listen to me. For the new PC-lint Plus product the cheapest license is a "Team of 1-5 developer" license at $4990. I will never understand why anybody would pick s licensing model which punishes one- or two-person companies, in an industry where there are so many of them. But it is Gimpels product and they get to make the rules. And me and my company gets to say "No way!" and keep the $1000 I would otherwise have sent their way. Licensing: ---------- If the four extra licenses could have been shared with four other developers in an Open Source Project, the pricing could have been made to work. But then we get to Gimpels licensing terms: Regarding Open Source You may use PC-lint Plus to analyze an open source project without other contributors being licensed as long as (1) you do not employ other developers to work on the project with you (in which case you would also need licenses for the developers you employ) and (2) you are not employed by another organization to work on the project (in which case all other developers employed by the same organization would also need to be licensed). I have no idea how Gimpel intends to count "developers" here? Does a PL/1-shark working on accounting on an IBM mainframe count ? What if he knows how to program C ? And in the specific case of Varnish Cache: What about VCL-developers ? I can't even imagine how Gimpel expects these restrictions to work, if some big software-house hires me for a single day to review a VMOD before they submit it to the Varnish Cache Open Source project ? Conclusion: ----------- No PC-lint Plus upgrade for me, I'll stick with my old FlexeLint. The good news is, that according to the many hours I spent testing the pre-release versions of PC-lint Plus, there is not that much performance difference from FlexeLint, and a fair number of bugs they need to fix to get the many false positives down. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.