From phk at FreeBSD.org Tue May 4 12:05:07 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 4 May 2021 12:05:07 +0000 (UTC) Subject: [master] 5bd17b74b Change acl::NO_FAM (which should never happen) from a VSL record to VRT_Fail Message-ID: <20210504120507.C5C1162716@lists.varnish-cache.org> commit 5bd17b74b2b09ee074a9fdd3882feaeb958e45aa Author: Poul-Henning Kamp Date: Tue May 4 11:35:30 2021 +0000 Change acl::NO_FAM (which should never happen) from a VSL record to VRT_Fail diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h index 94ed41409..2d3d3cf86 100644 --- a/include/tbl/vsl_tags.h +++ b/include/tbl/vsl_tags.h @@ -289,16 +289,15 @@ SLTM(Fetch_Body, 0, "Body fetched from backend", SLTM(VCL_acl, 0, "VCL ACL check results", "ACLs with the `+log` flag emits this record with the result.\n\n" "The format is::\n\n" - "\t%s [%s [%s [fixed: %s]]]\n" - "\t| | | |\n" - "\t| | | +- Fixed entry (see vcc_acl_pedantic parameter)\n" - "\t| | +------------ Matching entry (only for MATCH)\n" - "\t| +---------------- Name of the ACL for MATCH or NO_MATCH\n" - "\t+-------------------- MATCH, NO_MATCH or NO_FAM\n" + "\t%s %s [%s [fixed: %s]]\n" + "\t| | | |\n" + "\t| | | +- Fixed entry (see vcc_acl_pedantic parameter)\n" + "\t| | +------------ Matching entry (only for MATCH)\n" + "\t| +---------------- Name of the ACL\n" + "\t+-------------------- MATCH or NO_MATCH\n" "\n" "MATCH denotes an ACL match\n" "NO_MATCH denotes that a checked ACL has not matched\n" - "NO_FAM denotes a missing address family and should not occur.\n" "\n" ) diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c index f4a53a5aa..6f455ad5a 100644 --- a/lib/libvcc/vcc_acl.c +++ b/lib/libvcc/vcc_acl.c @@ -586,8 +586,8 @@ vcc_acl_emit(struct vcc *tl, const struct symbol *sym) Fh(tl, 0, "\n"); Fh(tl, 0, "\tfam = VRT_VSA_GetPtr(ctx, p, &a);\n"); Fh(tl, 0, "\tif (fam < 0) {\n"); - if (tl->acl->flag_log) - Fh(tl, 0, "\t\tVPI_acl_log(ctx, \"NO_FAM %s\");\n", sym->name); + Fh(tl, 0, "\t\tVRT_fail(ctx,"); + Fh(tl, 0, " \"ACL %s: no protocol family\");\n", sym->name); Fh(tl, 0, "\t\treturn(0);\n"); Fh(tl, 0, "\t}\n\n"); if (!tl->err_unref) { From phk at FreeBSD.org Tue May 4 12:05:07 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 4 May 2021 12:05:07 +0000 (UTC) Subject: [master] 65c9d4474 Make `pedantic` a per-acl feature flag, default to true. Message-ID: <20210504120507.DCF0462718@lists.varnish-cache.org> commit 65c9d447404a60a977afaffd0afdcd43b2e888ea Author: Poul-Henning Kamp Date: Tue May 4 12:03:02 2021 +0000 Make `pedantic` a per-acl feature flag, default to true. Also document `table` and `log` feature flags. Fixes #3269 diff --git a/bin/varnishtest/tests/c00005.vtc b/bin/varnishtest/tests/c00005.vtc index 6e8feaaa8..255f42232 100644 --- a/bin/varnishtest/tests/c00005.vtc +++ b/bin/varnishtest/tests/c00005.vtc @@ -72,7 +72,7 @@ varnish v1 -vcl { backend dummy None; - acl acl1 +log { + acl acl1 +log -pedantic { # bad notation (confusing) "1.2.3.4"/24; "1.2.3.66"/26; @@ -149,14 +149,12 @@ client c1 { logexpect l1 -wait -varnish v1 -cliok "param.set vcc_acl_pedantic on" - -varnish v1 -errvcl {Address/Netmask mismatch, need be 1.2.3.0/24} { +varnish v1 -errvcl {Non-zero bits in masked part} { import std; backend dummy None; - acl acl1 { + acl acl1 +pedantic { "1.2.3.4"/24; } diff --git a/bin/varnishtest/tests/m00023.vtc b/bin/varnishtest/tests/m00023.vtc index 749201696..3ecfbde70 100644 --- a/bin/varnishtest/tests/m00023.vtc +++ b/bin/varnishtest/tests/m00023.vtc @@ -12,7 +12,7 @@ varnish v1 -vcl+backend { "127"/24; } - acl locals { + acl locals -pedantic { // We assume c1 and s1 comes from same address "${s1_addr}"/24; } diff --git a/bin/varnishtest/tests/r01312.vtc b/bin/varnishtest/tests/r01312.vtc index ded2787e2..3fad2f0e5 100644 --- a/bin/varnishtest/tests/r01312.vtc +++ b/bin/varnishtest/tests/r01312.vtc @@ -40,11 +40,11 @@ varnish v1 -vcl+backend { import std; import debug; - acl foo { + acl foo -pedantic { "127.0.0.2"; "127.0.1"/19; } - acl bar { + acl bar -pedantic { "127.0.1.2"; "127.0.1"/19; } diff --git a/doc/changes.rst b/doc/changes.rst index 1fe26459f..7c82a0c3e 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -38,6 +38,11 @@ Varnish Cache 7.x.x (2021-09-15) * ACLs no longer produce VSL `VCL_acl` records by default, this must be explicitly enabled with `vcl +log { ... }`. +* ACLs can be compiled into a table format, which runs a little bit + slower, but compiles much faster for large ACLs. + +* ACLs default to `pedantic` which is no a per-ACL feature flag. + ================================ Varnish Cache 6.6.0 (2021-03-15) ================================ diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst index 3dbae425b..3513f0eb6 100644 --- a/doc/sphinx/reference/vcl.rst +++ b/doc/sphinx/reference/vcl.rst @@ -264,7 +264,8 @@ If an ACL entry specifies a host name which Varnish is unable to resolve, it will match any address it is compared to. Consequently, if it is preceded by a negation mark, it will reject any address it is compared to, which may not be what you intended. If the entry is -enclosed in parentheses, however, it will simply be ignored. +enclosed in parentheses, however, it will simply be ignored if the +host name cannot be resolved. To match an IP address against an ACL, simply use the match operator:: @@ -272,6 +273,24 @@ To match an IP address against an ACL, simply use the match operator:: return (pipe); } +ACLs have feature flags which can be set or cleared for each ACL +individually: + +* `+log` - Emit a `Acl` record in VSL to tell if a match was found + or not. + +* `+table` - Implement the ACL with a table instead of compiled code. + This runs a little bit slower, but compiles large ACLs much faster. + +* `-pedantic` - Allow masks to cover non-zero host-bits. + This allows the following to work:: + + acl foo -pedantic +log { + "firewall.example.com" / 24; + } + + However, if the name resolves to both IPv4 and IPv6 you will still + get an error. VCL objects ----------- diff --git a/include/tbl/mgt_vcc.h b/include/tbl/mgt_vcc.h index 9f828679a..908938367 100644 --- a/include/tbl/mgt_vcc.h +++ b/include/tbl/mgt_vcc.h @@ -3,7 +3,6 @@ * */ -MGT_VCC(unsigned, acl_pedantic, Acl_Pedantic) MGT_VCC(unsigned, allow_inline_c, Allow_InlineC) MGT_VCC(unsigned, err_unref, Err_Unref) MGT_VCC(unsigned, unsafe_path, Unsafe_Path) diff --git a/include/tbl/params.h b/include/tbl/params.h index cca420cb8..763920984 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -1588,22 +1588,6 @@ PARAM_VCC( "Unreferenced VCL objects result in error." ) -PARAM_VCC( - /* name */ vcc_acl_pedantic, - /* def */ "off", - /* descr */ - "Insist that network numbers used in ACLs have an " - "all-zero host part, e.g. make 1.2.3.4/24 an error.\n" - "With this option set to off (the default), the host " - "part of network numbers is being fixed to all-zeroes " - "(e.g. the above changed to 1.2.3.0/24), a warning is " - "output during VCL compilation and any ACL entry hits " - "are logged with the fixed address as \"fixed: ...\" " - "after the original VCL entry.\n" - "With this option set to on, any ACL entries with non-zero " - "host parts cause VCL compilation to fail." -) - PARAM_VCC( /* name */ vcc_allow_inline_c, /* def */ "off", diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c index 6f455ad5a..64dcc03ab 100644 --- a/lib/libvcc/vcc_acl.c +++ b/lib/libvcc/vcc_acl.c @@ -53,6 +53,7 @@ struct acl { #define VCC_ACL_MAGIC 0xb9fb3cd0 int flag_log; + int flag_pedantic; int flag_table; struct acl_tree acl_tree; @@ -201,14 +202,11 @@ vcc_acl_chk(struct vcc *tl, const struct acl_e *ae, const int l, AN(sa); VTCP_name(sa, h, sizeof h, NULL, 0); bprintf(t, "%s/%d", h, ae->mask); - VSB_cat(tl->sb, "Address/Netmask mismatch, "); - if (tl->acl_pedantic != 0) - VSB_printf(tl->sb, "need be %s\n", t); - else - VSB_printf(tl->sb, "changed to %s\n", t); - vcc_ErrWhere(tl, ae->t_addr); - if (tl->acl_pedantic == 0) - vcc_Warn(tl); + if (tl->acl->flag_pedantic != 0) { + VSB_cat(tl->sb, "Non-zero bits in masked part, "); + VSB_printf(tl->sb, "(maybe use %s ?)\n", t); + vcc_ErrWhere(tl, ae->t_addr); + } REPLACE(r, t); return (r); } @@ -697,6 +695,7 @@ vcc_ParseAcl(struct vcc *tl) INIT_OBJ(acl, VCC_ACL_MAGIC); tl->acl = acl; + acl->flag_pedantic = 1; vcc_NextToken(tl); VRBT_INIT(&acl->acl_tree); @@ -718,6 +717,9 @@ vcc_ParseAcl(struct vcc *tl) if (vcc_IdIs(tl->t, "log")) { acl->flag_log = sign; vcc_NextToken(tl); + } else if (vcc_IdIs(tl->t, "pedantic")) { + acl->flag_pedantic = sign; + vcc_NextToken(tl); } else if (vcc_IdIs(tl->t, "table")) { acl->flag_table = sign; vcc_NextToken(tl); From phk at FreeBSD.org Tue May 4 13:30:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 4 May 2021 13:30:06 +0000 (UTC) Subject: [master] eab1c394e Acl cannot be pedantic, localhost can be anything Message-ID: <20210504133006.3B83565238@lists.varnish-cache.org> commit eab1c394e59c3d270ac4ac56184c8233d61ad132 Author: Poul-Henning Kamp Date: Tue May 4 13:29:19 2021 +0000 Acl cannot be pedantic, localhost can be anything diff --git a/bin/varnishtest/tests/v00016.vtc b/bin/varnishtest/tests/v00016.vtc index c596e0170..f5145bf37 100644 --- a/bin/varnishtest/tests/v00016.vtc +++ b/bin/varnishtest/tests/v00016.vtc @@ -141,7 +141,7 @@ At:} { varnish v1 -syntax 4.0 -errvcl {Symbol not found: 'foo'} { backend b { .host = "${localhost}"; } - acl foo { + acl foo -pedantic { "${localhost}"/32; } sub vcl_init { From dridi.boukelmoune at gmail.com Wed May 5 06:38:07 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 5 May 2021 06:38:07 +0000 (UTC) Subject: [master] 525fe1808 doc: Typo Message-ID: <20210505063807.17FE8B164B@lists.varnish-cache.org> commit 525fe180846877dd3d36e7ca062755a756000c29 Author: Dridi Boukelmoune Date: Wed May 5 08:36:39 2021 +0200 doc: Typo diff --git a/doc/changes.rst b/doc/changes.rst index 7c82a0c3e..b903369cd 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -41,7 +41,7 @@ Varnish Cache 7.x.x (2021-09-15) * ACLs can be compiled into a table format, which runs a little bit slower, but compiles much faster for large ACLs. -* ACLs default to `pedantic` which is no a per-ACL feature flag. +* ACLs default to `pedantic` which is now a per-ACL feature flag. ================================ Varnish Cache 6.6.0 (2021-03-15) From phk at FreeBSD.org Wed May 5 08:07:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 5 May 2021 08:07:06 +0000 (UTC) Subject: [master] d4d7b385e Give struct worker a private stash for things vmods should not frob. Message-ID: <20210505080706.372516025@lists.varnish-cache.org> commit d4d7b385ef61f64cfcdfc5389a6437641d5317b6 Author: Poul-Henning Kamp Date: Wed May 5 08:06:06 2021 +0000 Give struct worker a private stash for things vmods should not frob. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index d5b7f458e..7c7e0a15e 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -87,23 +87,24 @@ enum { /*--------------------------------------------------------------------*/ -struct VSC_lck; -struct VSC_main; -struct VSC_main_wrk; struct ban; struct ban_proto; struct cli; struct http_conn; +struct listen_sock; struct mempool; struct objcore; struct objhead; struct pool; +struct req_step; struct sess; struct transport; -struct worker; -struct listen_sock; struct vcf; -struct req_step; +struct VSC_lck; +struct VSC_main; +struct VSC_main_wrk; +struct worker; +struct worker_priv; #define DIGEST_LEN 32 @@ -182,13 +183,6 @@ struct vsl_log { /*--------------------------------------------------------------------*/ -struct vxid_pool { - uint32_t next; - uint32_t count; -}; - -/*--------------------------------------------------------------------*/ - VRBT_HEAD(vrt_privs, vrt_priv); /* Worker pool stuff -------------------------------------------------*/ @@ -229,17 +223,15 @@ enum task_prio { struct worker { unsigned magic; #define WORKER_MAGIC 0x6391adcf + int strangelove; + struct worker_priv *wpriv; struct pool *pool; - struct objhead *nobjhead; - struct objcore *nobjcore; - void *nhashpriv; struct VSC_main_wrk *stats; struct vsl_log *vsl; // borrowed from req/bo struct pool_task task[1]; vtim_real lastused; - int strangelove; struct v1l *v1l; @@ -249,8 +241,6 @@ struct worker { struct ws aws[1]; - struct vxid_pool vxid_pool; - unsigned cur_method; unsigned seen_methods; unsigned handling; @@ -658,7 +648,7 @@ extern const char H__Reason[]; /* cache_main.c */ #define VXID(u) ((u) & VSL_IDENTMASK) -uint32_t VXID_Get(struct worker *, uint32_t marker); +uint32_t VXID_Get(const struct worker *, uint32_t marker); extern pthread_key_t witness_key; /* cache_lck.c */ diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index 5e07ccd46..c6c85120e 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -82,7 +82,7 @@ vbo_Free(struct busyobj **bop) } struct busyobj * -VBO_GetBusyObj(struct worker *wrk, const struct req *req) +VBO_GetBusyObj(const struct worker *wrk, const struct req *req) { struct busyobj *bo; uint16_t nhttp; diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index d94ec2ad5..2855c71e9 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -115,15 +115,15 @@ hsh_prealloc(struct worker *wrk) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); - if (wrk->nobjcore == NULL) - wrk->nobjcore = ObjNew(wrk); - CHECK_OBJ_NOTNULL(wrk->nobjcore, OBJCORE_MAGIC); + if (wrk->wpriv->nobjcore == NULL) + wrk->wpriv->nobjcore = ObjNew(wrk); + CHECK_OBJ_NOTNULL(wrk->wpriv->nobjcore, OBJCORE_MAGIC); - if (wrk->nobjhead == NULL) { - wrk->nobjhead = hsh_newobjhead(); + if (wrk->wpriv->nobjhead == NULL) { + wrk->wpriv->nobjhead = hsh_newobjhead(); wrk->stats->n_objecthead++; } - CHECK_OBJ_NOTNULL(wrk->nobjhead, OBJHEAD_MAGIC); + CHECK_OBJ_NOTNULL(wrk->wpriv->nobjhead, OBJHEAD_MAGIC); if (hash->prep != NULL) hash->prep(wrk); @@ -151,23 +151,26 @@ HSH_Private(const struct worker *wrk) } /*---------------------------------------------------------------------*/ + void -HSH_Cleanup(struct worker *wrk) +HSH_Cleanup(const struct worker *wrk) { - if (wrk->nobjcore != NULL) - ObjDestroy(wrk, &wrk->nobjcore); - - if (wrk->nobjhead != NULL) { - Lck_Delete(&wrk->nobjhead->mtx); - FREE_OBJ(wrk->nobjhead); - wrk->nobjhead = NULL; + CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(wrk->wpriv, WORKER_PRIV_MAGIC); + if (wrk->wpriv->nobjcore != NULL) + ObjDestroy(wrk, &wrk->wpriv->nobjcore); + + if (wrk->wpriv->nobjhead != NULL) { + Lck_Delete(&wrk->wpriv->nobjhead->mtx); + FREE_OBJ(wrk->wpriv->nobjhead); + wrk->wpriv->nobjhead = NULL; wrk->stats->n_objecthead--; } - if (wrk->nhashpriv != NULL) { + if (wrk->wpriv->nhashpriv != NULL) { /* XXX: If needed, add slinger method for this */ - free(wrk->nhashpriv); - wrk->nhashpriv = NULL; + free(wrk->wpriv->nhashpriv); + wrk->wpriv->nhashpriv = NULL; } } @@ -285,6 +288,7 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc, struct rush rush; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(wrk->wpriv, WORKER_PRIV_MAGIC); AN(digest); CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); AN(ban); @@ -295,8 +299,8 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc, hsh_prealloc(wrk); - AN(wrk->nobjhead); - oh = hash->lookup(wrk, digest, &wrk->nobjhead); + AN(wrk->wpriv->nobjhead); + oh = hash->lookup(wrk, digest, &wrk->wpriv->nobjhead); CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC); Lck_AssertHeld(&oh->mtx); assert(oh->refcnt > 0); @@ -329,15 +333,17 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc, */ static struct objcore * -hsh_insert_busyobj(struct worker *wrk, struct objhead *oh) +hsh_insert_busyobj(const struct worker *wrk, struct objhead *oh) { struct objcore *oc; + CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(wrk->wpriv, WORKER_PRIV_MAGIC); CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC); Lck_AssertHeld(&oh->mtx); - oc = wrk->nobjcore; - wrk->nobjcore = NULL; + oc = wrk->wpriv->nobjcore; + wrk->wpriv->nobjcore = NULL; CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); AN(oc->flags & OC_F_BUSY); @@ -372,6 +378,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp) CHECK_OBJ_NOTNULL(req, REQ_MAGIC); wrk = req->wrk; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); + CHECK_OBJ_NOTNULL(wrk->wpriv, WORKER_PRIV_MAGIC); CHECK_OBJ_NOTNULL(req->http, HTTP_MAGIC); CHECK_OBJ_ORNULL(req->vcf, VCF_MAGIC); AN(hash); @@ -390,8 +397,8 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp) Lck_Lock(&oh->mtx); req->hash_objhead = NULL; } else { - AN(wrk->nobjhead); - oh = hash->lookup(wrk, req->digest, &wrk->nobjhead); + AN(wrk->wpriv->nobjhead); + oh = hash->lookup(wrk, req->digest, &wrk->wpriv->nobjhead); } CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC); diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c index 1581f950d..a60e8a584 100644 --- a/bin/varnishd/cache/cache_main.c +++ b/bin/varnishd/cache/cache_main.c @@ -166,12 +166,13 @@ static uint32_t vxid_chunk = 32768; static struct lock vxid_lock; uint32_t -VXID_Get(struct worker *wrk, uint32_t mask) +VXID_Get(const struct worker *wrk, uint32_t mask) { struct vxid_pool *v; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); - v = &wrk->vxid_pool; + CHECK_OBJ_NOTNULL(wrk->wpriv, WORKER_PRIV_MAGIC); + v = wrk->wpriv->vxid_pool; AZ(VXID(mask)); do { if (v->count == 0) { diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 24eac15dc..7857b2f6b 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -1138,7 +1138,8 @@ CNT_Request(struct req *req) * pointers still pointing to the things we expect. */ CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); - CHECK_OBJ_ORNULL(wrk->nobjhead, OBJHEAD_MAGIC); + CHECK_OBJ_NOTNULL(wrk->wpriv, WORKER_PRIV_MAGIC); + CHECK_OBJ_ORNULL(wrk->wpriv->nobjhead, OBJHEAD_MAGIC); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); AN(req->req_step); @@ -1147,7 +1148,7 @@ CNT_Request(struct req *req) if (DO_DEBUG(DBG_REQ_STATE)) cnt_diag(req, req->req_step->name); nxt = req->req_step->func(wrk, req); - CHECK_OBJ_ORNULL(wrk->nobjhead, OBJHEAD_MAGIC); + CHECK_OBJ_ORNULL(wrk->wpriv->nobjhead, OBJHEAD_MAGIC); } wrk->vsl = NULL; if (nxt == REQ_FSM_DONE) { diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index f2e28b2a6..c9ccaabaf 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -65,6 +65,24 @@ struct req_step { extern const struct req_step R_STP_TRANSPORT[1]; extern const struct req_step R_STP_RECV[1]; +struct vxid_pool { + uint32_t next; + uint32_t count; +}; + +/*-------------------------------------------------------------------- + * Private part of worker threads + */ + +struct worker_priv { + unsigned magic; +#define WORKER_PRIV_MAGIC 0x3047db99 + struct objhead *nobjhead; + struct objcore *nobjcore; + void *nhashpriv; + struct vxid_pool vxid_pool[1]; +}; + /*-------------------------------------------------------------------- * HTTP Protocol connection structure * @@ -158,7 +176,7 @@ void BAN_RefBan(struct objcore *oc, struct ban *); vtim_real BAN_Time(const struct ban *ban); /* cache_busyobj.c */ -struct busyobj *VBO_GetBusyObj(struct worker *, const struct req *); +struct busyobj *VBO_GetBusyObj(const struct worker *, const struct req *); void VBO_ReleaseBusyObj(struct worker *wrk, struct busyobj **busyobj); /* cache_director.c */ diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index fcdab5d6f..a6962accb 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -87,6 +87,7 @@ wrk_bgthread(void *arg) { struct bgthread *bt; struct worker wrk; + struct worker_priv wpriv[1]; struct VSC_main_wrk ds; void *r; @@ -94,6 +95,8 @@ wrk_bgthread(void *arg) THR_SetName(bt->name); THR_Init(); INIT_OBJ(&wrk, WORKER_MAGIC); + INIT_OBJ(wpriv, WORKER_PRIV_MAGIC); + wrk.wpriv = wpriv; memset(&ds, 0, sizeof ds); wrk.stats = &ds; @@ -128,6 +131,7 @@ WRK_Thread(struct pool *qp, size_t stacksize, unsigned thread_workspace) struct worker *w, ww; struct VSC_main_wrk ds; unsigned char ws[thread_workspace]; + struct worker_priv wpriv[1]; AN(qp); AN(stacksize); @@ -136,6 +140,8 @@ WRK_Thread(struct pool *qp, size_t stacksize, unsigned thread_workspace) THR_SetName("cache-worker"); w = &ww; INIT_OBJ(w, WORKER_MAGIC); + INIT_OBJ(wpriv, WORKER_PRIV_MAGIC); + w->wpriv = wpriv; w->lastused = NAN; memset(&ds, 0, sizeof ds); w->stats = &ds; diff --git a/bin/varnishd/hash/hash_critbit.c b/bin/varnishd/hash/hash_critbit.c index 3d17a8451..c516a530c 100644 --- a/bin/varnishd/hash/hash_critbit.c +++ b/bin/varnishd/hash/hash_critbit.c @@ -192,8 +192,8 @@ hcb_crit_bit(const uint8_t *digest, const struct objhead *oh2, struct hcb_y *y) */ static struct objhead * -hcb_insert(struct worker *wrk, struct hcb_root *root, const uint8_t *digest, - struct objhead **noh) +hcb_insert(const struct worker *wrk, struct hcb_root *root, + const uint8_t *digest, struct objhead **noh) { volatile uintptr_t *p; uintptr_t pp; @@ -242,8 +242,8 @@ hcb_insert(struct worker *wrk, struct hcb_root *root, const uint8_t *digest, /* Insert */ - CAST_OBJ_NOTNULL(y2, wrk->nhashpriv, HCB_Y_MAGIC); - wrk->nhashpriv = NULL; + CAST_OBJ_NOTNULL(y2, wrk->wpriv->nhashpriv, HCB_Y_MAGIC); + wrk->wpriv->nhashpriv = NULL; (void)hcb_crit_bit(digest, oh2, y2); s2 = (digest[y2->ptr] & y2->bitmask) != 0; assert(s2 < 2); @@ -405,7 +405,7 @@ hcb_lookup(struct worker *wrk, const void *digest, struct objhead **noh) while (1) { /* No luck, try with lock held, so we can modify tree */ - CAST_OBJ_NOTNULL(y, wrk->nhashpriv, HCB_Y_MAGIC); + CAST_OBJ_NOTNULL(y, wrk->wpriv->nhashpriv, HCB_Y_MAGIC); Lck_Lock(&hcb_mtx); VSC_C_main->hcb_lock++; oh = hcb_insert(wrk, &hcb_root, digest, noh); @@ -440,10 +440,10 @@ hcb_prep(struct worker *wrk) { struct hcb_y *y; - if (wrk->nhashpriv == NULL) { + if (wrk->wpriv->nhashpriv == NULL) { ALLOC_OBJ(y, HCB_Y_MAGIC); AN(y); - wrk->nhashpriv = y; + wrk->wpriv->nhashpriv = y; } } diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h index 66d4c2639..a1a9c0e82 100644 --- a/bin/varnishd/hash/hash_slinger.h +++ b/bin/varnishd/hash/hash_slinger.h @@ -66,7 +66,7 @@ void HSH_config(const char *); /* cache_hash.c */ void HSH_Init(const struct hash_slinger *); -void HSH_Cleanup(struct worker *); +void HSH_Cleanup(const struct worker *); extern const struct hash_slinger hsl_slinger; extern const struct hash_slinger hcl_slinger; From phk at FreeBSD.org Wed May 5 09:28:04 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 5 May 2021 09:28:04 +0000 (UTC) Subject: [master] f195de9bf Keep a local 'now' and reference Lck_CondWait to it. Message-ID: <20210505092804.BD45696CD@lists.varnish-cache.org> commit f195de9bf2bc772058696c779085244d2e6d74b4 Author: Poul-Henning Kamp Date: Wed May 5 09:23:58 2021 +0000 Keep a local 'now' and reference Lck_CondWait to it. Without this fix workers could busy-spin when idle under debug+vtc_mode. diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index a6962accb..bdb8d5734 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -375,7 +375,7 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk) { struct pool_task *tp; struct pool_task tpx, tps; - vtim_real tmo; + vtim_real tmo, now; unsigned i, reserve; CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); @@ -420,6 +420,7 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk) wrk->task->priv = wrk; VTAILQ_INSERT_HEAD(&pp->idle_queue, wrk->task, list); pp->nidle++; + now = wrk->lastused; do { // see signaling_note at the top for explanation if (DO_DEBUG(DBG_VCLREL) && @@ -431,16 +432,14 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk) * stats. Set a 1 second timeout * so that we'll wake up and get a * chance to push stats. */ - tmo = wrk->lastused + 1.; + tmo = now + 1.; else if (wrk->vcl == NULL) tmo = 0; else if (DO_DEBUG(DBG_VTC_MODE)) - tmo = wrk->lastused+1.; + tmo = now + 1.; else - tmo = wrk->lastused+60.; + tmo = now + 60.; i = Lck_CondWait(&wrk->cond, &pp->mtx, tmo); - if (i == ETIMEDOUT && wrk->vcl != NULL) - VCL_Rel(&wrk->vcl); if (wrk->task->func != NULL) { /* We have been handed a new task */ tpx = *wrk->task; @@ -465,6 +464,13 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk) pp->a_stat = pp->b_stat; pp->b_stat = NULL; tp = &tps; + } else { + // Presumably ETIMEDOUT but we do not + // assert this because pthread condvars + // are not airtight. + if (wrk->vcl) + VCL_Rel(&wrk->vcl); + now = VTIM_real(); } } while (tp == NULL); } From phk at FreeBSD.org Wed May 5 17:37:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 5 May 2021 17:37:06 +0000 (UTC) Subject: [master] e3a6652c9 If we have a vcl on req0, recache it, it's almost certain to be vcl_active. Message-ID: <20210505173706.4C8E49D5D4@lists.varnish-cache.org> commit e3a6652c9c796fc6df415b21c71f63ecd59cd48a Author: Poul-Henning Kamp Date: Wed May 5 17:35:57 2021 +0000 If we have a vcl on req0, recache it, it's almost certain to be vcl_active. diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 7857b2f6b..63531cd81 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -1157,7 +1157,7 @@ CNT_Request(struct req *req) if (IS_TOPREQ(req)) { VCL_TaskLeave(ctx, req->top->privs); if (req->top->vcl0 != NULL) - VCL_Rel(&req->top->vcl0); + VCL_Recache(wrk, &req->top->vcl0); } VCL_TaskLeave(ctx, req->privs); AN(req->vsl->wid); From phk at FreeBSD.org Wed May 5 17:46:04 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 5 May 2021 17:46:04 +0000 (UTC) Subject: [master] 9ed99f61e Move the worker vcl cache to the private part Message-ID: <20210505174604.BEC769DB3C@lists.varnish-cache.org> commit 9ed99f61ebecb8471e87f3d6bdfde34b9937929d Author: Poul-Henning Kamp Date: Wed May 5 17:44:55 2021 +0000 Move the worker vcl cache to the private part diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 7c7e0a15e..2ea190180 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -237,8 +237,6 @@ struct worker { pthread_cond_t cond; - struct vcl *vcl; - struct ws aws[1]; unsigned cur_method; diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index 8ee346cba..a57978e6a 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -445,8 +445,8 @@ vca_accept_task(struct worker *wrk, void *arg) VTIM_sleep(.1); /* Dont hold on to (possibly) discarded VCLs */ - if (wrk->vcl != NULL) - VCL_Rel(&wrk->vcl); + if (wrk->wpriv->vcl != NULL) + VCL_Rel(&wrk->wpriv->vcl); while (!ps->pool->die) { INIT_OBJ(&wa, WRK_ACCEPT_MAGIC); diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 63531cd81..0d26f4615 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -1097,9 +1097,9 @@ CNT_Embark(struct worker *wrk, struct req *req) req->vfc->wrk = req->wrk = wrk; wrk->vsl = req->vsl; if (req->req_step == R_STP_TRANSPORT && req->vcl == NULL) { - VCL_Refresh(&wrk->vcl); - req->vcl = wrk->vcl; - wrk->vcl = NULL; + VCL_Refresh(&wrk->wpriv->vcl); + req->vcl = wrk->wpriv->vcl; + wrk->wpriv->vcl = NULL; VSLb(req->vsl, SLT_VCL_use, "%s", VCL_Name(req->vcl)); } diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index c9ccaabaf..96db999b1 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -81,6 +81,7 @@ struct worker_priv { struct objcore *nobjcore; void *nhashpriv; struct vxid_pool vxid_pool[1]; + struct vcl *vcl; }; /*-------------------------------------------------------------------- @@ -480,7 +481,7 @@ void VCL_VRT_Init(void); const char *VCL_Return_Name(unsigned); const char *VCL_Method_Name(unsigned); void VCL_Refresh(struct vcl **); -void VCL_Recache(struct worker *, struct vcl **); +void VCL_Recache(const struct worker *, struct vcl **); void VCL_Ref(struct vcl *); void VCL_Rel(struct vcl **); VCL_BACKEND VCL_DefaultDirector(const struct vcl *); diff --git a/bin/varnishd/cache/cache_vrt_vcl.c b/bin/varnishd/cache/cache_vrt_vcl.c index a862163ef..9dcd5a55b 100644 --- a/bin/varnishd/cache/cache_vrt_vcl.c +++ b/bin/varnishd/cache/cache_vrt_vcl.c @@ -91,20 +91,20 @@ VCL_Refresh(struct vcl **vcc) } void -VCL_Recache(struct worker *wrk, struct vcl **vclp) +VCL_Recache(const struct worker *wrk, struct vcl **vclp) { AN(wrk); AN(vclp); CHECK_OBJ_NOTNULL(*vclp, VCL_MAGIC); - if (*vclp != vcl_active || wrk->vcl == vcl_active) { + if (*vclp != vcl_active || wrk->wpriv->vcl == vcl_active) { VCL_Rel(vclp); return; } - if (wrk->vcl != NULL) - VCL_Rel(&wrk->vcl); - wrk->vcl = *vclp; + if (wrk->wpriv->vcl != NULL) + VCL_Rel(&wrk->wpriv->vcl); + wrk->wpriv->vcl = *vclp; *vclp = NULL; } diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index bdb8d5734..6b4825633 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -155,8 +155,8 @@ WRK_Thread(struct pool *qp, size_t stacksize, unsigned thread_workspace) AZ(w->pool); VSL(SLT_WorkThread, 0, "%p end", w); - if (w->vcl != NULL) - VCL_Rel(&w->vcl); + if (w->wpriv->vcl != NULL) + VCL_Rel(&w->wpriv->vcl); AZ(pthread_cond_destroy(&w->cond)); HSH_Cleanup(w); Pool_Sumstat(w); @@ -433,7 +433,7 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk) * so that we'll wake up and get a * chance to push stats. */ tmo = now + 1.; - else if (wrk->vcl == NULL) + else if (wrk->wpriv->vcl == NULL) tmo = 0; else if (DO_DEBUG(DBG_VTC_MODE)) tmo = now + 1.; @@ -468,8 +468,8 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk) // Presumably ETIMEDOUT but we do not // assert this because pthread condvars // are not airtight. - if (wrk->vcl) - VCL_Rel(&wrk->vcl); + if (wrk->wpriv->vcl) + VCL_Rel(&wrk->wpriv->vcl); now = VTIM_real(); } } while (tp == NULL); @@ -483,8 +483,8 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk) memset(wrk->task, 0, sizeof wrk->task); assert(wrk->pool == pp); tp->func(wrk, tp->priv); - if (DO_DEBUG(DBG_VCLREL) && wrk->vcl != NULL) - VCL_Rel(&wrk->vcl); + if (DO_DEBUG(DBG_VCLREL) && wrk->wpriv->vcl != NULL) + VCL_Rel(&wrk->wpriv->vcl); tpx = *wrk->task; tp = &tpx; } while (tp->func != NULL); diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 26207a3bc..88bc74a95 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -347,8 +347,8 @@ h2_new_session(struct worker *wrk, void *arg) sp = req->sp; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); - if (wrk->vcl) - VCL_Rel(&wrk->vcl); + if (wrk->wpriv->vcl) + VCL_Rel(&wrk->wpriv->vcl); assert(req->transport == &H2_transport); From dridi at varni.sh Wed May 5 19:10:43 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 5 May 2021 19:10:43 +0000 Subject: [master] 96b7bf526 Make the VTIM_format() test 64bit only In-Reply-To: <20210430202610.37DF295E8F@lists.varnish-cache.org> References: <20210430202610.37DF295E8F@lists.varnish-cache.org> Message-ID: On Fri, Apr 30, 2021 at 8:26 PM Nils Goroll wrote: > > > commit 96b7bf5265fe9a22f920fbe70d034ae57f0ae1c3 > Author: Nils Goroll > Date: Fri Apr 30 22:19:47 2021 +0200 > > Make the VTIM_format() test 64bit only > > It is not obvious how we could trigger a gmtime_r() EOVERFLOW on 32bit > > Thank you phk for pushing me back onto the right track at a late hour. > > Ref #3308 Unironically caught by ubsan complaining about a float cast overflow. From phk at FreeBSD.org Thu May 6 06:11:07 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 6 May 2021 06:11:07 +0000 (UTC) Subject: [master] d55f0f0ab Rescue the testcase from #3193 Message-ID: <20210506061107.ADC906410@lists.varnish-cache.org> commit d55f0f0ab9baa5c1f7ece34b43cf8e87bc0ebca1 Author: Poul-Henning Kamp Date: Thu May 6 06:10:21 2021 +0000 Rescue the testcase from #3193 diff --git a/bin/varnishtest/tests/c00053.vtc b/bin/varnishtest/tests/c00053.vtc index 6a8e50bb1..20a5f1854 100644 --- a/bin/varnishtest/tests/c00053.vtc +++ b/bin/varnishtest/tests/c00053.vtc @@ -1,4 +1,4 @@ -varnishtest "Test include vs. unsafe_path" +varnishtest "Test include vs. unsafe_path and include glob-ing" server s1 { rxreq @@ -25,3 +25,45 @@ varnish v1 -vcl+backend { } shell "rm -f ${tmpdir}/_.c00053" + +# Testing of include +glob + +varnish v1 -cliok "param.set vcc_unsafe_path on" + +varnish v1 -errvcl "glob pattern matched no files." { + vcl 4.0; + + include +glob "${tmpdir}/Q*.vcl"; +} + +shell { + echo 'sub vcl_deliver { set resp.http.foo = "foo"; }' > ${tmpdir}/sub_foo.vcl + echo 'sub vcl_deliver { set resp.http.bar = "bar"; }' > ${tmpdir}/sub_bar.vcl + echo 'vcl 4.0; backend default { .host = "0:0"; } include +glob "./sub_*.vcl";' > ${tmpdir}/top.vcl +} + +varnish v1 -vcl+backend { + include +glob "${tmpdir}/sub_*.vcl"; +} -start + +client c1 { + txreq + rxresp + expect resp.http.foo == foo + expect resp.http.bar == bar +} -run + +varnish v1 -errvcl {needs absolute filename of including file.} { + include +glob "./sub_*.vcl"; + backend default none; +} + +varnish v1 -cliok "vcl.load foo ${tmpdir}/top.vcl" +varnish v1 -cliok "vcl.use foo" + +client c1 { + txreq + rxresp + expect resp.http.foo == foo + expect resp.http.bar == bar +} -run From phk at FreeBSD.org Thu May 6 07:21:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 6 May 2021 07:21:05 +0000 (UTC) Subject: [master] d042154ee Let jailed glob(3) read ${tmpdir} Message-ID: <20210506072105.72F9C974F@lists.varnish-cache.org> commit d042154eefdf59f1fa7ca5c0e24b1b71ff37241c Author: Poul-Henning Kamp Date: Thu May 6 07:20:42 2021 +0000 Let jailed glob(3) read ${tmpdir} diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c index 31f8cb712..309bd876e 100644 --- a/bin/varnishtest/vtc_main.c +++ b/bin/varnishtest/vtc_main.c @@ -407,7 +407,7 @@ start_test(void) bprintf(tmpdir, "%s/vtc.%d.%08x", tmppath, (int)getpid(), (unsigned)random()); - AZ(mkdir(tmpdir, 0711)); + AZ(mkdir(tmpdir, 0755)); tp = VTAILQ_FIRST(&tst_head); CHECK_OBJ_NOTNULL(tp, TST_MAGIC); From dridi.boukelmoune at gmail.com Thu May 6 17:31:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 6 May 2021 17:31:05 +0000 (UTC) Subject: [master] 9f8a3cf4a vcl: Centralize struct vcldir tear down Message-ID: <20210506173105.A3509A208A@lists.varnish-cache.org> commit 9f8a3cf4af9114114ccdfc6af2e3ab7611cad251 Author: Dridi Boukelmoune Date: Thu May 6 19:26:21 2021 +0200 vcl: Centralize struct vcldir tear down And doing so plug the director lock leak spotted by asan. Refs 47586588fccde1869a8bec76001406fc6ebb0b84 Refs 6e9b8d6d7ca93d365914c044307fab455d4ec750 diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index 2ecbc8c3c..d70e9543f 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -457,11 +457,9 @@ vcl_KillBackends(struct vcl *vcl) if (vdir == NULL) break; VTAILQ_REMOVE(&vcl->director_list, vdir, list); - REPLACE(vdir->cli_name, NULL); AN(vdir->methods->destroy); vdir->methods->destroy(vdir->dir); - FREE_OBJ(vdir->dir); - FREE_OBJ(vdir); + vcldir_free(vdir); } Lck_Unlock(&vcl_mtx); } diff --git a/bin/varnishd/cache/cache_vcl.h b/bin/varnishd/cache/cache_vcl.h index 885e918a8..0c72c22ec 100644 --- a/bin/varnishd/cache/cache_vcl.h +++ b/bin/varnishd/cache/cache_vcl.h @@ -71,6 +71,7 @@ extern struct lock vcl_mtx; extern struct vcl *vcl_active; /* protected by vcl_mtx */ struct vcl *vcl_find(const char *); void VCL_Update(struct vcl **, struct vcl *); +void vcldir_free(struct vcldir *); struct vcltemp { const char * const name; diff --git a/bin/varnishd/cache/cache_vrt_vcl.c b/bin/varnishd/cache/cache_vrt_vcl.c index 9dcd5a55b..9b3c25a0f 100644 --- a/bin/varnishd/cache/cache_vrt_vcl.c +++ b/bin/varnishd/cache/cache_vrt_vcl.c @@ -138,7 +138,7 @@ VCL_Rel(struct vcl **vcc) /*--------------------------------------------------------------------*/ -static void +void vcldir_free(struct vcldir *vdir) { From phk at FreeBSD.org Fri May 7 08:56:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 7 May 2021 08:56:05 +0000 (UTC) Subject: [master] 3bf8fc777 Refactoring Message-ID: <20210507085605.DD5CB64FA8@lists.varnish-cache.org> commit 3bf8fc777bedb677d3a8427b62ab97ac82546619 Author: Poul-Henning Kamp Date: Thu May 6 07:38:06 2021 +0000 Refactoring diff --git a/lib/libvcc/vcc_source.c b/lib/libvcc/vcc_source.c index 521b5b51e..272e9d7cb 100644 --- a/lib/libvcc/vcc_source.c +++ b/lib/libvcc/vcc_source.c @@ -159,105 +159,116 @@ vcc_include_glob_file(struct vcc *tl, const struct source *src_sp, } /*-------------------------------------------------------------------- - * NB: We cannot use vcc_ErrWhere2() on tokens which are no on the + * NB: We cannot use vcc_ErrWhere2() on tokens which are not on the * NB: tl->tokens list. */ -void -vcc_lex_source(struct vcc *tl, struct source *src_sp, int eoi) +static struct token * +vcc_lex_include(struct vcc *tl, const struct source *src_sp, struct token *t) { - struct token *t, *tok1; + struct token *tok1; int i, glob_flag = 0; const struct source *sp1; struct vsb *vsb = NULL; const char *filename; const char *p; - vcc_Lexer(tl, src_sp); - if (tl->err) - return; - VTAILQ_FOREACH(t, &src_sp->src_tokens, src_list) { - if (!eoi && t->tok == EOI) - break; + assert(vcc_IdIs(t, "include")); - if (t->tok != ID || !vcc_IdIs(t, "include")) { - VTAILQ_INSERT_TAIL(&tl->tokens, t, list); - continue; - } + tok1 = VTAILQ_NEXT(t, src_list); + AN(tok1); + while (1) { + t = VTAILQ_NEXT(tok1, src_list); + AN(t); + i = vcc_IsFlagRaw(tl, tok1, t); + if (i < 0) + break; + if (vcc_IdIs(t, "glob")) { + glob_flag = i; + } else { + VSB_cat(tl->sb, "Unknown include flag:\n"); + vcc_ErrWhere(tl, t); + return(t); + } tok1 = VTAILQ_NEXT(t, src_list); AN(tok1); + } - while (1) { - t = VTAILQ_NEXT(tok1, src_list); - AN(t); - i = vcc_IsFlagRaw(tl, tok1, t); - if (i < 0) - break; - if (vcc_IdIs(t, "glob")) { - glob_flag = i; - } else { - VSB_cat(tl->sb, "Unknown include flag:\n"); - vcc_ErrWhere(tl, t); - return; - } - tok1 = VTAILQ_NEXT(t, src_list); - AN(tok1); - } + if (tok1->tok != CSTR) { + VSB_cat(tl->sb, + "include not followed by string constant.\n"); + vcc_ErrWhere(tl, tok1); + return (t); + } + t = VTAILQ_NEXT(tok1, src_list); + AN(t); - if (tok1->tok != CSTR) { - VSB_cat(tl->sb, - "include not followed by string constant.\n"); - vcc_ErrWhere(tl, tok1); - return; - } - t = VTAILQ_NEXT(tok1, src_list); - AN(t); + if (t->tok != ';') { + VSB_cat(tl->sb, + "include not followed by semicolon.\n"); + vcc_ErrWhere(tl, tok1); + return (t); + } - if (t->tok != ';') { + filename = tok1->dec; + + if (filename[0] == '.' && filename[1] == '/') { + /* + * Nested include filenames, starting with "./" are + * resolved relative to the VCL file which contains + * the include directive. + */ + if (src_sp->name[0] != '/') { VSB_cat(tl->sb, - "include not followed by semicolon.\n"); + "include \"./xxxxx\"; needs absolute " + "filename of including file.\n"); vcc_ErrWhere(tl, tok1); - return; + return(t); } + vsb = VSB_new_auto(); + AN(vsb); + p = strrchr(src_sp->name, '/'); + AN(p); + VSB_bcat(vsb, src_sp->name, p - src_sp->name); + VSB_cat(vsb, filename + 1); + AZ(VSB_finish(vsb)); + filename = VSB_data(vsb); + } - filename = tok1->dec; - - if (filename[0] == '.' && filename[1] == '/') { - /* - * Nested include filenames, starting with "./" are - * resolved relative to the VCL file which contains - * the include directive. - */ - if (src_sp->name[0] != '/') { - VSB_cat(tl->sb, - "include \"./xxxxx\"; needs absolute " - "filename of including file.\n"); - return; - } - vsb = VSB_new_auto(); - AN(vsb); - p = strrchr(src_sp->name, '/'); - AN(p); - VSB_bcat(vsb, src_sp->name, p - src_sp->name); - VSB_cat(vsb, filename + 1); - AZ(VSB_finish(vsb)); - filename = VSB_data(vsb); + if (glob_flag) + i = vcc_include_glob_file(tl, src_sp, filename, tok1); + else + i = vcc_include_file(tl, src_sp, filename, tok1); + if (vsb != NULL) + VSB_destroy(&vsb); + if (i) { + vcc_ErrWhere(tl, tok1); + for (sp1 = src_sp; sp1 != NULL; sp1 = sp1->parent) { + if (sp1->parent_tok == NULL) + break; + vcc_ErrWhere(tl, sp1->parent_tok); } + } + return (t); +} - if (glob_flag) - i = vcc_include_glob_file(tl, src_sp, filename, tok1); - else - i = vcc_include_file(tl, src_sp, filename, tok1); - if (vsb != NULL) - VSB_destroy(&vsb); - if (i) { - vcc_ErrWhere(tl, tok1); - for (sp1 = src_sp; sp1 != NULL; sp1 = sp1->parent) { - if (sp1->parent_tok == NULL) - break; - vcc_ErrWhere(tl, sp1->parent_tok); - } +void +vcc_lex_source(struct vcc *tl, struct source *src_sp, int eoi) +{ + struct token *t; + + vcc_Lexer(tl, src_sp); + if (tl->err) + return; + VTAILQ_FOREACH(t, &src_sp->src_tokens, src_list) { + if (!eoi && t->tok == EOI) + break; + + if (t->tok == ID && vcc_IdIs(t, "include")) { + t = vcc_lex_include(tl, src_sp, t); + } else { + VTAILQ_INSERT_TAIL(&tl->tokens, t, list); } if (tl->err) return; From phk at FreeBSD.org Fri May 7 08:56:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 7 May 2021 08:56:05 +0000 (UTC) Subject: [master] a911a136e Refactoring Message-ID: <20210507085605.F257264FAB@lists.varnish-cache.org> commit a911a136e86bd46219c41f7811a2b667dda75581 Author: Poul-Henning Kamp Date: Thu May 6 08:51:04 2021 +0000 Refactoring diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 7d01ae9c8..2219531d1 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -433,6 +433,8 @@ void vcc_Duration(struct vcc *tl, double *); unsigned vcc_UintVal(struct vcc *tl); int vcc_IsFlag(struct vcc *tl); int vcc_IsFlagRaw(struct vcc *, const struct token *, const struct token *); +char *vcc_Dup_be(const char *b, const char *e); +int vcc_Has_vcl_prefix(const char *b); /* vcc_var.c */ sym_wildcard_t vcc_Var_Wildcard; diff --git a/lib/libvcc/vcc_parse.c b/lib/libvcc/vcc_parse.c index 41fe8dd03..432d5681b 100644 --- a/lib/libvcc/vcc_parse.c +++ b/lib/libvcc/vcc_parse.c @@ -244,10 +244,7 @@ vcc_ParseFunction(struct vcc *tl) p = sym->proc; if (p == NULL) { if (vcc_builtin != NULL && bsym == NULL && - (t->b[0] == 'v'|| t->b[0] == 'V') && - (t->b[1] == 'c'|| t->b[1] == 'C') && - (t->b[2] == 'l'|| t->b[2] == 'L') && - (t->b[3] == '_')) { + vcc_Has_vcl_prefix(t->b)) { VSB_printf(tl->sb,"The names 'vcl_*'" " are reserved for subroutines.\n"); vcc_ErrWhere(tl, t); diff --git a/lib/libvcc/vcc_symb.c b/lib/libvcc/vcc_symb.c index 0423c3c14..6e8d275b5 100644 --- a/lib/libvcc/vcc_symb.c +++ b/lib/libvcc/vcc_symb.c @@ -125,22 +125,6 @@ VCC_SymName(struct vsb *vsb, const struct symbol *sym) vcc_symtabname(vsb, sym->symtab); } -static char * -vcc_dup_be(const char *b, const char *e) -{ - char *p; - - AN(b); - if (e == NULL) - e = strchr(b, '\0'); - AN(e); - assert(e >= b); - - p = strndup(b, e - b); - AN(p); - return (p); -} - static struct symtab * vcc_symtab_new(const char *name) { @@ -177,14 +161,14 @@ vcc_symtab_str(struct symtab *st, const char *b, const char *e) continue; if (i == 0 && l == st2->nlen) break; - st3 = vcc_symtab_new(vcc_dup_be(b, q)); + st3 = vcc_symtab_new(vcc_Dup_be(b, q)); st3->parent = st; VTAILQ_INSERT_BEFORE(st2, st3, list); st2 = st3; break; } if (st2 == NULL) { - st2 = vcc_symtab_new(vcc_dup_be(b, q)); + st2 = vcc_symtab_new(vcc_Dup_be(b, q)); st2->parent = st; VTAILQ_INSERT_TAIL(&st->children, st2, list); } @@ -287,10 +271,7 @@ VCC_SymbolGet(struct vcc *tl, vcc_ns_t ns, vcc_kind_t kind, AN(x); AN(x->name); if (tl->syntax >= VCL_41 && e == SYMTAB_CREATE && kind != SYM_SUB && - (tl->t->b[0] == 'v'|| tl->t->b[0] == 'V') && - (tl->t->b[1] == 'c'|| tl->t->b[1] == 'C') && - (tl->t->b[2] == 'l'|| tl->t->b[2] == 'L') && - (tl->t->b[3] == '_')) { + vcc_Has_vcl_prefix(tl->t->b)) { VSB_cat(tl->sb, "Symbols named 'vcl_*' are reserved.\nAt:"); vcc_ErrWhere(tl, tl->t); return (NULL); diff --git a/lib/libvcc/vcc_utils.c b/lib/libvcc/vcc_utils.c index d60d932cd..7e4e00a99 100644 --- a/lib/libvcc/vcc_utils.c +++ b/lib/libvcc/vcc_utils.c @@ -385,3 +385,30 @@ vcc_IsFlag(struct vcc *tl) vcc_NextToken(tl); return (retval); } + +char * +vcc_Dup_be(const char *b, const char *e) +{ + char *p; + + AN(b); + if (e == NULL) + e = strchr(b, '\0'); + AN(e); + assert(e >= b); + + p = strndup(b, e - b); + AN(p); + return (p); +} + +int +vcc_Has_vcl_prefix(const char *b) +{ + return ( + (b[0] == 'v' || b[0] == 'V') && + (b[1] == 'c' || b[1] == 'C') && + (b[2] == 'l' || b[2] == 'L') && + (b[3] == '_') + ); +} From phk at FreeBSD.org Fri May 7 08:56:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 7 May 2021 08:56:06 +0000 (UTC) Subject: [master] bbaf8aa79 Refactor Message-ID: <20210507085606.1B37864FAE@lists.varnish-cache.org> commit bbaf8aa79d8e434391006b783e25a44eceb68d64 Author: Poul-Henning Kamp Date: Thu May 6 10:11:25 2021 +0000 Refactor diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index 88d558a93..a82fc9760 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -636,8 +636,6 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile) /* Register and lex the main source */ if (sp != NULL) { AN(vcc_builtin); - VTAILQ_INSERT_TAIL(&tl->sources, sp, list); - sp->idx = tl->nsources++; vcc_lex_source(tl, sp, 0); if (tl->err) return (NULL); @@ -646,8 +644,6 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile) /* Register and lex the builtin VCL */ sp = vcc_new_source(tl->builtin_vcl, "Builtin"); assert(sp != NULL); - VTAILQ_INSERT_TAIL(&tl->sources, sp, list); - sp->idx = tl->nsources++; vcc_lex_source(tl, sp, 1); if (tl->err) return (NULL); diff --git a/lib/libvcc/vcc_source.c b/lib/libvcc/vcc_source.c index 272e9d7cb..db953d2be 100644 --- a/lib/libvcc/vcc_source.c +++ b/lib/libvcc/vcc_source.c @@ -121,8 +121,6 @@ vcc_include_file(struct vcc *tl, const struct source *src_sp, return (-1); } sp->parent = src_sp; - VTAILQ_INSERT_TAIL(&tl->sources, sp, list); - sp->idx = tl->nsources++; sp->parent_tok = parent_token; vcc_lex_source(tl, sp, 0); return (0); @@ -258,6 +256,9 @@ vcc_lex_source(struct vcc *tl, struct source *src_sp, int eoi) { struct token *t; + VTAILQ_INSERT_TAIL(&tl->sources, src_sp, list); + src_sp->idx = tl->nsources++; + vcc_Lexer(tl, src_sp); if (tl->err) return; From phk at FreeBSD.org Fri May 7 08:56:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 7 May 2021 08:56:06 +0000 (UTC) Subject: [master] 05192a0e5 Also test recursion through include +glob Message-ID: <20210507085606.34BCE64FB7@lists.varnish-cache.org> commit 05192a0e5fd813fd5a65f826beb0db952b13ac50 Author: Poul-Henning Kamp Date: Fri May 7 08:24:25 2021 +0000 Also test recursion through include +glob diff --git a/bin/varnishtest/tests/r03360.vtc b/bin/varnishtest/tests/r03360.vtc index 9c1e81c10..337ce7c69 100644 --- a/bin/varnishtest/tests/r03360.vtc +++ b/bin/varnishtest/tests/r03360.vtc @@ -1,7 +1,7 @@ varnishtest "Test recusive vcl includes" shell {echo include '"_recurse.vcl";' > ${tmpdir}/_recurse.vcl} -shell {echo include '"_recurse2.vcl";' > ${tmpdir}/_recurse1.vcl} +shell {echo include +glob '"${tmpdir}/_recurse[2].vcl";' > ${tmpdir}/_recurse1.vcl} shell {echo include '"_recurse1.vcl";' > ${tmpdir}/_recurse2.vcl} varnish v1 -arg "-p vcl_path=${tmpdir}" -errvcl "Recursive include of" { From phk at FreeBSD.org Fri May 7 08:56:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 7 May 2021 08:56:06 +0000 (UTC) Subject: [master] ed278561f Refactor the recursive include check. Message-ID: <20210507085606.4F78664FBA@lists.varnish-cache.org> commit ed278561f1e4c77d3968814e9ece9015b148d3cc Author: Poul-Henning Kamp Date: Fri May 7 08:53:51 2021 +0000 Refactor the recursive include check. diff --git a/bin/varnishtest/tests/r03360.vtc b/bin/varnishtest/tests/r03360.vtc index 337ce7c69..94f9fd21c 100644 --- a/bin/varnishtest/tests/r03360.vtc +++ b/bin/varnishtest/tests/r03360.vtc @@ -4,12 +4,12 @@ shell {echo include '"_recurse.vcl";' > ${tmpdir}/_recurse.vcl} shell {echo include +glob '"${tmpdir}/_recurse[2].vcl";' > ${tmpdir}/_recurse1.vcl} shell {echo include '"_recurse1.vcl";' > ${tmpdir}/_recurse2.vcl} -varnish v1 -arg "-p vcl_path=${tmpdir}" -errvcl "Recursive include of" { +varnish v1 -arg "-p vcl_path=${tmpdir}" -errvcl "Recursive use of file" { backend b { .host = "${localhost}"; } include "_recurse.vcl" ; } -varnish v2 -arg "-p vcl_path=${tmpdir}" -errvcl "Recursive include of" { +varnish v2 -arg "-p vcl_path=${tmpdir}" -errvcl "Recursive use of file" { backend b { .host = "${localhost}"; } include "_recurse1.vcl" ; } diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index a82fc9760..d270b0ef1 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -642,7 +642,7 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile) } /* Register and lex the builtin VCL */ - sp = vcc_new_source(tl->builtin_vcl, "Builtin"); + sp = vcc_new_source(tl->builtin_vcl, "builtin", ""); assert(sp != NULL); vcc_lex_source(tl, sp, 1); if (tl->err) @@ -790,7 +790,7 @@ VCC_Compile(struct vcc *tl, struct vsb **sb, } if (vclsrc != NULL) - sp = vcc_new_source(vclsrc, vclsrcfile); + sp = vcc_new_source(vclsrc, "vcl.inline", vclsrcfile); else sp = vcc_file_source(tl, vclsrcfile); diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 2219531d1..58308d951 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -79,12 +79,14 @@ void vcl_output_lang_h(struct vsb *sb); #define INDENT 2 struct source { + unsigned magic; +#define SOURCE_MAGIC 0xf756fe82 VTAILQ_ENTRY(source) list; + const char *kind; char *name; const char *b; const char *e; unsigned idx; - char *freeit; const struct source *parent; const struct token *parent_tok; VTAILQ_HEAD(, token) src_tokens; @@ -364,8 +366,9 @@ void vcc_Parse_Init(struct vcc *); sym_act_f vcc_Act_If; /* vcc_source.c */ -struct source * vcc_new_source(const char *src, const char *name); -struct source *vcc_file_source(const struct vcc *tl, const char *fn); +struct source * vcc_new_source(const char *src, const char *kind, + const char *name); +struct source *vcc_file_source(struct vcc *tl, const char *fn); void vcc_lex_source(struct vcc *tl, struct source *sp, int eoi); /* vcc_storage.c */ diff --git a/lib/libvcc/vcc_source.c b/lib/libvcc/vcc_source.c index db953d2be..57167b37e 100644 --- a/lib/libvcc/vcc_source.c +++ b/lib/libvcc/vcc_source.c @@ -41,15 +41,16 @@ #include "vfil.h" struct source * -vcc_new_source(const char *src, const char *name) +vcc_new_source(const char *src, const char *kind, const char *name) { struct source *sp; AN(src); AN(name); - sp = calloc(1, sizeof *sp); + ALLOC_OBJ(sp, SOURCE_MAGIC); AN(sp); REPLACE(sp->name, name); + sp->kind = kind; sp->b = src; sp->e = strchr(src, '\0'); VTAILQ_INIT(&sp->src_tokens); @@ -58,31 +59,15 @@ vcc_new_source(const char *src, const char *name) /*--------------------------------------------------------------------*/ -static void -vcc_destroy_source(struct source **spp) -{ - struct source *sp; - - AN(spp); - sp = *spp; - *spp = NULL; - - AN(sp); - free(sp->name); - free(sp->freeit); - free(sp); -} - -/*--------------------------------------------------------------------*/ - struct source * -vcc_file_source(const struct vcc *tl, const char *fn) +vcc_file_source(struct vcc *tl, const char *fn) { char *f, *fnp; struct source *sp; if (!tl->unsafe_path && strchr(fn, '/') != NULL) { VSB_printf(tl->sb, "VCL filename '%s' is unsafe.\n", fn); + tl->err = 1; return (NULL); } f = NULL; @@ -90,70 +75,60 @@ vcc_file_source(const struct vcc *tl, const char *fn) VSB_printf(tl->sb, "Cannot read file '%s' (%s)\n", fnp != NULL ? fnp : fn, strerror(errno)); free(fnp); + tl->err = 1; return (NULL); } - sp = vcc_new_source(f, fnp); + sp = vcc_new_source(f, "file", fnp); free(fnp); - sp->freeit = f; return (sp); } /*--------------------------------------------------------------------*/ -static int +static void vcc_include_file(struct vcc *tl, const struct source *src_sp, const char *filename, const struct token *parent_token) { struct source *sp; - const struct source *sp1; sp = vcc_file_source(tl, filename); if (sp == NULL) - return (-1); + return; - for (sp1 = src_sp; sp1 != NULL; sp1 = sp1->parent) - if (!strcmp(sp1->name, sp->name)) - break; - if (sp1 != NULL) { - VSB_printf(tl->sb, - "Recursive include of \"%s\"\n\n", sp->name); - vcc_destroy_source(&sp); - return (-1); - } sp->parent = src_sp; sp->parent_tok = parent_token; vcc_lex_source(tl, sp, 0); - return (0); } /*--------------------------------------------------------------------*/ -static int +static void vcc_include_glob_file(struct vcc *tl, const struct source *src_sp, const char *filename, const struct token *parent_token) { glob_t g[1]; - int rv; unsigned u; + int i; memset(g, 0, sizeof g); - rv = glob(filename, 0, NULL, g); - switch (rv) { + i = glob(filename, 0, NULL, g); + switch (i) { case 0: - for (u = 0; rv == 0 && u < g->gl_pathc; u++) { - rv = vcc_include_file( + for (u = 0; !tl->err && u < g->gl_pathc; u++) { + vcc_include_file( tl, src_sp, g->gl_pathv[u], parent_token); } break; case GLOB_NOMATCH: VSB_printf(tl->sb, "glob pattern matched no files.\n"); + tl->err = 1; break; default: - VSB_printf(tl->sb, "glob(3) expansion failed (%d)\n", rv); + VSB_printf(tl->sb, "glob(3) expansion failed (%d)\n", i); + tl->err = 1; break; } globfree(g); - return (rv); } /*-------------------------------------------------------------------- @@ -166,7 +141,6 @@ vcc_lex_include(struct vcc *tl, const struct source *src_sp, struct token *t) { struct token *tok1; int i, glob_flag = 0; - const struct source *sp1; struct vsb *vsb = NULL; const char *filename; const char *p; @@ -235,19 +209,13 @@ vcc_lex_include(struct vcc *tl, const struct source *src_sp, struct token *t) } if (glob_flag) - i = vcc_include_glob_file(tl, src_sp, filename, tok1); + vcc_include_glob_file(tl, src_sp, filename, tok1); else - i = vcc_include_file(tl, src_sp, filename, tok1); + vcc_include_file(tl, src_sp, filename, tok1); if (vsb != NULL) VSB_destroy(&vsb); - if (i) { + if (tl->err) vcc_ErrWhere(tl, tok1); - for (sp1 = src_sp; sp1 != NULL; sp1 = sp1->parent) { - if (sp1->parent_tok == NULL) - break; - vcc_ErrWhere(tl, sp1->parent_tok); - } - } return (t); } @@ -255,6 +223,20 @@ void vcc_lex_source(struct vcc *tl, struct source *src_sp, int eoi) { struct token *t; + const struct source *sp1; + + CHECK_OBJ_NOTNULL(src_sp, SOURCE_MAGIC); + + for (sp1 = src_sp->parent; sp1 != NULL; sp1 = sp1->parent) { + if (!strcmp(sp1->name, src_sp->name) && + !strcmp(sp1->kind, src_sp->kind)) { + VSB_printf(tl->sb, + "Recursive use of %s \"%s\"\n\n", + src_sp->kind, src_sp->name); + tl->err = 1; + return; + } + } VTAILQ_INSERT_TAIL(&tl->sources, src_sp, list); src_sp->idx = tl->nsources++; From phk at FreeBSD.org Fri May 7 09:32:04 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 7 May 2021 09:32:04 +0000 (UTC) Subject: [master] 8a1948fa2 BLOBs have a .tostring now, but test other type which do not. Message-ID: <20210507093204.EAA516E90C@lists.varnish-cache.org> commit 8a1948fa2b11a8f9d4ba2f81bf719f60ade3da36 Author: Poul-Henning Kamp Date: Fri May 7 09:16:05 2021 +0000 BLOBs have a .tostring now, but test other type which do not. diff --git a/bin/varnishtest/tests/v00020.vtc b/bin/varnishtest/tests/v00020.vtc index 68570d106..ac60663aa 100644 --- a/bin/varnishtest/tests/v00020.vtc +++ b/bin/varnishtest/tests/v00020.vtc @@ -388,6 +388,10 @@ varnish v1 -errvcl {Symbol 'acl' has wrong type (reserved), expected acl:} { sub vcl_recv { if (client.ip ~ acl) {} } } +varnish v1 -errvcl {Cannot convert HTTP to STRING} { + sub vcl_synth { set resp.http.foo = resp; } +} + server s1 { rxreq txresp -hdr "bar: X" diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 9b30c438a..3a762a1ee 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -288,15 +288,9 @@ vcc_expr_tostring(struct vcc *tl, struct expr **e, vcc_type_t fmt) (*e)->constant = constant; (*e)->nstr = 1; } else { - if ((*e)->fmt == BLOB) - VSB_cat(tl->sb, - "Wrong use of BLOB value.\n" - "BLOBs can only be used as arguments to VMOD" - " functions.\n"); - else - VSB_printf(tl->sb, - "Cannot convert %s to STRING.\n", - vcc_utype((*e)->fmt)); + VSB_printf(tl->sb, + "Cannot convert %s to STRING.\n", + vcc_utype((*e)->fmt)); vcc_ErrWhere2(tl, (*e)->t1, tl->t); } } From phk at FreeBSD.org Fri May 7 09:32:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 7 May 2021 09:32:05 +0000 (UTC) Subject: [master] 7aef0530c More vcc_expr coverage (CBLOBs) Message-ID: <20210507093205.0CE4F6E911@lists.varnish-cache.org> commit 7aef0530cf52567b22e5e2a96e2e321b3d5c498e Author: Poul-Henning Kamp Date: Fri May 7 09:21:49 2021 +0000 More vcc_expr coverage (CBLOBs) diff --git a/bin/varnishtest/tests/v00020.vtc b/bin/varnishtest/tests/v00020.vtc index ac60663aa..f437e92a2 100644 --- a/bin/varnishtest/tests/v00020.vtc +++ b/bin/varnishtest/tests/v00020.vtc @@ -342,6 +342,13 @@ varnish v1 -errvcl {INT * BLOB not possible.} { } } +varnish v1 -errvcl {INT * BLOB not possible.} { + import blob; + sub vcl_deliver { + set resp.status = 100 * :4thASR0O18ZxnoKtc4zd8KuO25rPvwvMQyAvRfilz6o=:); + } +} + # XXX: should spot nonexistent storage varnish v1 -errvcl {Symbol not found: 'storage.foo'} { sub vcl_backend_response { From phk at FreeBSD.org Fri May 7 09:36:04 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 7 May 2021 09:36:04 +0000 (UTC) Subject: [master] dd9d7927d More vcc_expr.c coverage Message-ID: <20210507093604.9B2126EE5F@lists.varnish-cache.org> commit dd9d7927dc7e44dd061f68ddf6307a2fe10a9161 Author: Poul-Henning Kamp Date: Fri May 7 09:35:34 2021 +0000 More vcc_expr.c coverage diff --git a/bin/varnishtest/tests/v00020.vtc b/bin/varnishtest/tests/v00020.vtc index f437e92a2..d5df60d4a 100644 --- a/bin/varnishtest/tests/v00020.vtc +++ b/bin/varnishtest/tests/v00020.vtc @@ -395,6 +395,11 @@ varnish v1 -errvcl {Symbol 'acl' has wrong type (reserved), expected acl:} { sub vcl_recv { if (client.ip ~ acl) {} } } +varnish v1 -errvcl {Symbol 'default' is a reserved word.} { + import std; + sub vcl_recv { set req.http.foo = default; } +} + varnish v1 -errvcl {Cannot convert HTTP to STRING} { sub vcl_synth { set resp.http.foo = resp; } } From nils.goroll at uplex.de Fri May 7 13:52:05 2021 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 May 2021 13:52:05 +0000 (UTC) Subject: [master] a9ac9f010 Do not dereference cached pointers in the PRIV_TASK fini method Message-ID: <20210507135205.417DF9BB8A@lists.varnish-cache.org> commit a9ac9f0103b7bc2ab4a7059c9cda8495faf5b3bf Author: Nils Goroll Date: Fri May 7 15:25:10 2021 +0200 Do not dereference cached pointers in the PRIV_TASK fini method The main discussion of this topic is going to happen in #3600, but whatever the outcome, the fini method does not happen within the task, so we must not dereference any cached pointers (unless we own them). Closes #3606 diff --git a/bin/varnishtest/tests/v00041.vtc b/bin/varnishtest/tests/v00041.vtc index d7cf4652f..5995d82a4 100644 --- a/bin/varnishtest/tests/v00041.vtc +++ b/bin/varnishtest/tests/v00041.vtc @@ -121,8 +121,8 @@ logexpect l0 -v v1 -g raw -d 1 -m -q "vxid == 0" { expect 0 = Debug {^objb.priv_task.. = .*"initY"} expect 0 = VCL_Log {^objb initY} expect ? = Debug {^priv_task_fini} - expect ? = Debug {^obj_priv_task_fini.*"initX"} - expect ? = Debug {^obj_priv_task_fini.*"initY"} + expect ? = Debug {^obj_priv_task_fini} + expect ? = Debug {^obj_priv_task_fini} expect 0 = Debug {^vcl1: VCL_EVENT_WARM} # need an anchor for the ? expects to begin @@ -174,7 +174,7 @@ logexpect l1001 -v v1 -g vxid -q "vxid == 1001" { expect 0 = VCL_return {^deliver} expect 9 = Timestamp {^Resp} expect ? = Debug {^priv_task_fini} - expect ? = Debug {^obj_priv_task_fini.*"d1001"} + expect ? = Debug {^obj_priv_task_fini} } -start logexpect l1002 -v v1 -g vxid -q "vxid == 1002" { @@ -203,7 +203,7 @@ logexpect l1002 -v v1 -g vxid -q "vxid == 1002" { expect 0 = VCL_return {^deliver} expect 9 = Timestamp {^BerespBody} expect ? = Debug {^priv_task_fini} - expect ? = Debug {^obj_priv_task_fini.*"r1002"} + expect ? = Debug {^obj_priv_task_fini} } -start logexpect l1006 -v v1 -g vxid -q "vxid == 1006" { @@ -237,7 +237,7 @@ logexpect l1006 -v v1 -g vxid -q "vxid == 1006" { expect 0 = VCL_return {^pipe} expect 4 = PipeAcct expect ? = Debug {^priv_task_fini} - expect ? = Debug {^obj_priv_task_fini.*"p1006"} + expect ? = Debug {^obj_priv_task_fini} } -start client c1 { diff --git a/vmod/vmod_debug_obj.c b/vmod/vmod_debug_obj.c index cca01c1db..f5e92a2e5 100644 --- a/vmod/vmod_debug_obj.c +++ b/vmod/vmod_debug_obj.c @@ -157,11 +157,11 @@ xyzzy_obj_test_priv_vcl(VRT_CTX, static void v_matchproto_(vmod_priv_fini_f) \ obj_priv_ ## name ## _fini(VRT_CTX, void *ptr) \ { \ - const char * const fmt = "obj_priv_" #name "_fini(%p = \"%s\")"; \ + const char * const fmt = "obj_priv_" #name "_fini(%p)"; \ \ CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); \ AN(ptr); \ - mylog(ctx->vsl, SLT_Debug, fmt, ptr, (char *)ptr); \ + mylog(ctx->vsl, SLT_Debug, fmt, ptr); \ } \ \ static const struct vmod_priv_methods \ From dridi.boukelmoune at gmail.com Mon May 10 13:14:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 13:14:06 +0000 (UTC) Subject: [master] 90623cc5c vtim: Avoid a float cast overflow Message-ID: <20210510131406.40E29A1CC3@lists.varnish-cache.org> commit 90623cc5cdb220df78b476c77c6e1c50295e979b Author: Dridi Boukelmoune Date: Wed May 5 21:36:03 2021 +0200 vtim: Avoid a float cast overflow Spotted by ubsan. diff --git a/lib/libvarnish/vtim.c b/lib/libvarnish/vtim.c index 676330e1b..1ca0d836d 100644 --- a/lib/libvarnish/vtim.c +++ b/lib/libvarnish/vtim.c @@ -60,6 +60,7 @@ #include #include +#include #include #include #include @@ -168,15 +169,20 @@ VTIM_format(vtim_real t, char *p) time_t tt; AN(p); - tt = (time_t) t; - if (gmtime_r(&tt, &tm) != NULL) - AN(snprintf(p, VTIM_FORMAT_SIZE, - "%s, %02d %s %4d %02d:%02d:%02d GMT", - weekday_name[tm.tm_wday], - tm.tm_mday, month_name[tm.tm_mon], - tm.tm_year + 1900, tm.tm_hour, tm.tm_min, tm.tm_sec)); - else - *p = '\0'; + *p = '\0'; + + if (t < (vtim_real)INTMAX_MIN || t > (vtim_real)INTMAX_MAX) + return; + + tt = (time_t)(intmax_t)t; + if (gmtime_r(&tt, &tm) == NULL) + return; + + AN(snprintf(p, VTIM_FORMAT_SIZE, + "%s, %02d %s %4d %02d:%02d:%02d GMT", + weekday_name[tm.tm_wday], + tm.tm_mday, month_name[tm.tm_mon], tm.tm_year + 1900, + tm.tm_hour, tm.tm_min, tm.tm_sec)); } #ifdef TEST_DRIVER From dridi.boukelmoune at gmail.com Mon May 10 13:14:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 13:14:06 +0000 (UTC) Subject: [master] a99e3f94f style: Whitespace OCD Message-ID: <20210510131406.57494A1CC7@lists.varnish-cache.org> commit a99e3f94f00d465d7e53f45ac465916f6d631e80 Author: Dridi Boukelmoune Date: Mon May 10 14:21:50 2021 +0200 style: Whitespace OCD diff --git a/lib/libvcc/vcc_symb.c b/lib/libvcc/vcc_symb.c index 6e8d275b5..e2cda32b3 100644 --- a/lib/libvcc/vcc_symb.c +++ b/lib/libvcc/vcc_symb.c @@ -236,24 +236,28 @@ const struct symxref XREF_DEF[1] = {{"xref_def"}}; const struct symxref XREF_REF[1] = {{"xref_ref"}}; const struct symmode SYMTAB_NOERR[1] = {{ - .name = "sym_noerror", - .noerr = 1 - }}; + .name = "sym_noerror", + .noerr = 1 +}}; + const struct symmode SYMTAB_CREATE[1] = {{ - .name = "sym_create" - }}; + .name = "sym_create" +}}; + const struct symmode SYMTAB_EXISTING[1] = {{ - .name = "Symbol not found" - }}; + .name = "Symbol not found" +}}; + const struct symmode SYMTAB_PARTIAL[1] = {{ - .name = "Symbol not found", - .partial = 1 - }}; + .name = "Symbol not found", + .partial = 1 +}}; + const struct symmode SYMTAB_PARTIAL_NOERR[1] = {{ - .name = "Symbol not found", - .partial = 1, - .noerr = 1 - }}; + .name = "Symbol not found", + .partial = 1, + .noerr = 1 +}}; struct symbol * VCC_SymbolGet(struct vcc *tl, vcc_ns_t ns, vcc_kind_t kind, diff --git a/vmod/vmod_cookie.c b/vmod/vmod_cookie.c index 2cb1605e8..a0f47c74b 100644 --- a/vmod/vmod_cookie.c +++ b/vmod/vmod_cookie.c @@ -81,9 +81,9 @@ cobj_free(VRT_CTX, void *p) } static const struct vmod_priv_methods cookie_cobj_priv_methods[1] = {{ - .magic = VMOD_PRIV_METHODS_MAGIC, - .type = "vmod_cookie_cobj", - .fini = cobj_free + .magic = VMOD_PRIV_METHODS_MAGIC, + .type = "vmod_cookie_cobj", + .fini = cobj_free }}; static struct vmod_cookie * diff --git a/vmod/vmod_debug.c b/vmod/vmod_debug.c index 8723481eb..fa75dd130 100644 --- a/vmod/vmod_debug.c +++ b/vmod/vmod_debug.c @@ -242,10 +242,10 @@ priv_ ## name ## _fini(VRT_CTX, void *ptr) \ \ static const struct vmod_priv_methods \ xyzzy_test_priv_ ## name ## _methods[1] = {{ \ - .magic = VMOD_PRIV_METHODS_MAGIC, \ - .type = "debug_test_priv_" #name, \ - .fini = priv_ ## name ## _fini \ - }}; + .magic = VMOD_PRIV_METHODS_MAGIC, \ + .type = "debug_test_priv_" #name, \ + .fini = priv_ ## name ## _fini \ +}}; PRIV_FINI(call, 0) PRIV_FINI(task, 1) PRIV_FINI(top, 1) @@ -426,9 +426,9 @@ priv_vcl_fini(VRT_CTX, void *priv) } static const struct vmod_priv_methods priv_vcl_methods[1] = {{ - .magic = VMOD_PRIV_METHODS_MAGIC, - .type = "debug_priv_vcl_fini", - .fini = priv_vcl_fini + .magic = VMOD_PRIV_METHODS_MAGIC, + .type = "debug_priv_vcl_fini", + .fini = priv_vcl_fini }}; static int @@ -1201,9 +1201,9 @@ fail_f(VRT_CTX, void *priv) } static const struct vmod_priv_methods xyzzy_fail_task_fini_methods[1] = {{ - .magic = VMOD_PRIV_METHODS_MAGIC, - .type = "debug_fail_task_fini", - .fini = fail_f + .magic = VMOD_PRIV_METHODS_MAGIC, + .type = "debug_fail_task_fini", + .fini = fail_f }}; VCL_VOID v_matchproto_(td_xyzzy_debug_fail_task_fini) diff --git a/vmod/vmod_directors_shard_cfg.c b/vmod/vmod_directors_shard_cfg.c index 6ba55b692..6cd709df8 100644 --- a/vmod/vmod_directors_shard_cfg.c +++ b/vmod/vmod_directors_shard_cfg.c @@ -100,9 +100,9 @@ shard_change_fini(VRT_CTX, void * priv) } static const struct vmod_priv_methods shard_change_priv_methods[1] = {{ - .magic = VMOD_PRIV_METHODS_MAGIC, - .type = "vmod_directors_shard_cfg", - .fini = shard_change_fini + .magic = VMOD_PRIV_METHODS_MAGIC, + .type = "vmod_directors_shard_cfg", + .fini = shard_change_fini }}; static struct shard_change * diff --git a/vmod/vmod_std_fileread.c b/vmod/vmod_std_fileread.c index bd0f0fbc8..b77b1e727 100644 --- a/vmod/vmod_std_fileread.c +++ b/vmod/vmod_std_fileread.c @@ -85,9 +85,9 @@ fini_frfile(VRT_CTX, void *ptr) } static const struct vmod_priv_methods frfile_methods[1] = {{ - .magic = VMOD_PRIV_METHODS_MAGIC, - .type = "vmod_std_fileread", - .fini = fini_frfile + .magic = VMOD_PRIV_METHODS_MAGIC, + .type = "vmod_std_fileread", + .fini = fini_frfile }}; static struct frfile * From dridi.boukelmoune at gmail.com Mon May 10 13:14:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 13:14:06 +0000 (UTC) Subject: [master] 9df3808ea build: Pass sanitizer flags to cc_command Message-ID: <20210510131406.78354A1CCB@lists.varnish-cache.org> commit 9df3808eae81d6d033d124e64ed7706c55eb94b1 Author: Dridi Boukelmoune Date: Mon May 10 14:36:04 2021 +0200 build: Pass sanitizer flags to cc_command To get the same checks for the code we generate with libvcc. diff --git a/configure.ac b/configure.ac index de24d8ceb..3e02d06d0 100644 --- a/configure.ac +++ b/configure.ac @@ -802,7 +802,7 @@ else VCC_CC="exec cc $OCFLAGS -dynamiclib -Wl,-undefined,dynamic_lookup -o %o %s" ;; *) - VCC_CC="exec $PTHREAD_CC $OCFLAGS $PTHREAD_CFLAGS -fpic -shared -Wl,-x -o %o %s" + VCC_CC="exec $PTHREAD_CC $OCFLAGS $PTHREAD_CFLAGS $SAN_CFLAGS -fpic -shared -Wl,-x -o %o %s" ;; esac fi From dridi.boukelmoune at gmail.com Mon May 10 13:14:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 13:14:06 +0000 (UTC) Subject: [master] db556310d vtc.7: Polish Message-ID: <20210510131406.928F5A1CD3@lists.varnish-cache.org> commit db556310d0ad7c03fce3a204dae36aefad678477 Author: Dridi Boukelmoune Date: Mon May 10 14:44:01 2021 +0200 vtc.7: Polish diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c index be7236b7e..88c78982d 100644 --- a/bin/varnishtest/vtc_misc.c +++ b/bin/varnishtest/vtc_misc.c @@ -398,9 +398,9 @@ ipvx_works(const char *target) * 64bit * The environment is 64 bits * ipv4 - * 127.0.0.1 work + * 127.0.0.1 works * ipv6 - * [::1] work + * [::1] works * dns * DNS lookups are working * topbuild From dridi.boukelmoune at gmail.com Mon May 10 13:14:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 13:14:06 +0000 (UTC) Subject: [master] 79797c666 vtc: New feature check no_sanitizer Message-ID: <20210510131406.B4D5AA1CDC@lists.varnish-cache.org> commit 79797c66666a1d5b6a902bc4a954bce117484c75 Author: Dridi Boukelmoune Date: Mon May 10 14:45:10 2021 +0200 vtc: New feature check no_sanitizer diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c index 88c78982d..53e50812f 100644 --- a/bin/varnishtest/vtc_misc.c +++ b/bin/varnishtest/vtc_misc.c @@ -420,6 +420,8 @@ ipvx_works(const char *target) * recognized as a macro. * persistent_storage * Varnish was built with the deprecated persistent storage. + * no_sanitizer + * Varnish was not built with a sanitizer. * * Be careful with ignore_unknown_macro, because it may cause a test with a * misspelled macro to fail silently. You should only need it if you must @@ -432,6 +434,12 @@ static const unsigned with_persistent_storage = 1; static const unsigned with_persistent_storage = 0; #endif +#if __SANITIZER +static const unsigned no_sanitizer = 0; +#else +static const unsigned no_sanitizer = 1; +#endif + void v_matchproto_(cmd_f) cmd_feature(CMD_ARGS) { @@ -475,6 +483,7 @@ cmd_feature(CMD_ARGS) FEATURE("user_vcache", getpwnam("vcache") != NULL); FEATURE("group_varnish", getgrnam("varnish") != NULL); FEATURE("persistent_storage", with_persistent_storage); + FEATURE("no_sanitizer", no_sanitizer); if (!strcmp(*av, "disable_aslr")) { good = 1; From dridi.boukelmoune at gmail.com Mon May 10 13:14:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 13:14:06 +0000 (UTC) Subject: [master] d2e71a9b0 vtc: Skip v4 if we have sanitizers Message-ID: <20210510131406.D5820A1CE9@lists.varnish-cache.org> commit d2e71a9b011ffc17afe0292193ff6ae4376c4fbd Author: Dridi Boukelmoune Date: Mon May 10 14:50:49 2021 +0200 vtc: Skip v4 if we have sanitizers The sanity check of our defaults shouldn't be affected by sanitizer overhead. diff --git a/bin/varnishtest/tests/v00004.vtc b/bin/varnishtest/tests/v00004.vtc index a5345b9f7..2f2b75459 100644 --- a/bin/varnishtest/tests/v00004.vtc +++ b/bin/varnishtest/tests/v00004.vtc @@ -1,6 +1,7 @@ varnishtest "test if our default paramers make sense ..." feature 64bit +feature no_sanitizer # Skip this test in GCOV mode, 68xx bytes extra per level makes it fail feature cmd {test -z "$GCOVPROG"} From dridi.boukelmoune at gmail.com Mon May 10 13:14:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 13:14:06 +0000 (UTC) Subject: [master] 74ee676c0 vtc: Simplify SO_RCVTIMEO_WORKS feature check Message-ID: <20210510131406.EF331A1CF0@lists.varnish-cache.org> commit 74ee676c0c4a4a5cf9ccc828fcc9584448f3d58d Author: Dridi Boukelmoune Date: Mon May 10 14:52:23 2021 +0200 vtc: Simplify SO_RCVTIMEO_WORKS feature check diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c index 53e50812f..63cddfdbf 100644 --- a/bin/varnishtest/vtc_misc.c +++ b/bin/varnishtest/vtc_misc.c @@ -440,6 +440,12 @@ static const unsigned no_sanitizer = 0; static const unsigned no_sanitizer = 1; #endif +#ifdef SO_RCVTIMEO_WORKS +static const unsigned so_rcvtimeo_works = 1; +#else +static const unsigned so_rcvtimeo_works = 0; +#endif + void v_matchproto_(cmd_f) cmd_feature(CMD_ARGS) { @@ -464,13 +470,6 @@ cmd_feature(CMD_ARGS) for (av++; *av != NULL; av++) { good = 0; - if (!strcmp(*av, "SO_RCVTIMEO_WORKS")) { -#ifdef SO_RCVTIMEO_WORKS - good = 1; -#else - vtc_stop = 2; -#endif - } FEATURE("ipv4", ipvx_works("127.0.0.1")); FEATURE("ipv6", ipvx_works("[::1]")); @@ -484,6 +483,7 @@ cmd_feature(CMD_ARGS) FEATURE("group_varnish", getgrnam("varnish") != NULL); FEATURE("persistent_storage", with_persistent_storage); FEATURE("no_sanitizer", no_sanitizer); + FEATURE("SO_RCVTIMEO_WORKS", so_rcvtimeo_works); if (!strcmp(*av, "disable_aslr")) { good = 1; From dridi.boukelmoune at gmail.com Mon May 10 13:14:07 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 13:14:07 +0000 (UTC) Subject: [master] bf3912448 lck: Make it clear that we take ilck away Message-ID: <20210510131407.22B48A1CFA@lists.varnish-cache.org> commit bf3912448125e62702ea4696910f9df20c6b0f9b Author: Dridi Boukelmoune Date: Mon May 10 15:08:05 2021 +0200 lck: Make it clear that we take ilck away diff --git a/bin/varnishd/cache/cache_lck.c b/bin/varnishd/cache/cache_lck.c index c3c726708..3e507119c 100644 --- a/bin/varnishd/cache/cache_lck.c +++ b/bin/varnishd/cache/cache_lck.c @@ -269,9 +269,8 @@ Lck_Delete(struct lock *lck) { struct ilck *ilck; - CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC); + TAKE_OBJ_NOTNULL(ilck, &lck->priv, ILCK_MAGIC); ilck->stat->destroy++; - lck->priv = NULL; AZ(pthread_mutex_destroy(&ilck->mtx)); FREE_OBJ(ilck); } From dridi.boukelmoune at gmail.com Mon May 10 15:24:04 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 10 May 2021 15:24:04 +0000 (UTC) Subject: [master] 929e94b5c lck: Check lock pointers before dereferencing them Message-ID: <20210510152404.D4C46A78AE@lists.varnish-cache.org> commit 929e94b5c5516d06d3d8d92fe70d0c32f91f9303 Author: Dridi Boukelmoune Date: Mon May 10 17:17:52 2021 +0200 lck: Check lock pointers before dereferencing them diff --git a/bin/varnishd/cache/cache_lck.c b/bin/varnishd/cache/cache_lck.c index 3e507119c..32b326ee1 100644 --- a/bin/varnishd/cache/cache_lck.c +++ b/bin/varnishd/cache/cache_lck.c @@ -115,6 +115,7 @@ Lck__Lock(struct lock *lck, const char *p, int l) struct ilck *ilck; int r = EINVAL; + AN(lck); CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC); if (DO_DEBUG(DBG_WITNESS)) Lck_Witness_Lock(ilck, p, l, ""); @@ -140,6 +141,7 @@ Lck__Unlock(struct lock *lck, const char *p, int l) (void)p; (void)l; + AN(lck); CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC); assert(pthread_equal(ilck->owner, pthread_self())); AN(ilck->held); @@ -170,6 +172,7 @@ Lck__Trylock(struct lock *lck, const char *p, int l) struct ilck *ilck; int r; + AN(lck); CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC); if (DO_DEBUG(DBG_WITNESS)) Lck_Witness_Lock(ilck, p, l, "?"); @@ -190,6 +193,7 @@ Lck__Held(const struct lock *lck) { struct ilck *ilck; + AN(lck); CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC); return (ilck->held); } @@ -199,6 +203,7 @@ Lck__Owned(const struct lock *lck) { struct ilck *ilck; + AN(lck); CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC); AN(ilck->held); return (pthread_equal(ilck->owner, pthread_self())); @@ -211,6 +216,7 @@ Lck_CondWait(pthread_cond_t *cond, struct lock *lck, vtim_real when) struct timespec ts; vtim_real t; + AN(lck); CAST_OBJ_NOTNULL(ilck, lck->priv, ILCK_MAGIC); AN(ilck->held); assert(pthread_equal(ilck->owner, pthread_self())); @@ -254,6 +260,7 @@ Lck__New(struct lock *lck, struct VSC_lck *st, const char *w) AN(st); AN(w); + AN(lck); AZ(lck->priv); ALLOC_OBJ(ilck, ILCK_MAGIC); AN(ilck); @@ -269,6 +276,7 @@ Lck_Delete(struct lock *lck) { struct ilck *ilck; + AN(lck); TAKE_OBJ_NOTNULL(ilck, &lck->priv, ILCK_MAGIC); ilck->stat->destroy++; AZ(pthread_mutex_destroy(&ilck->mtx)); From phk at FreeBSD.org Mon May 10 18:08:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 May 2021 18:08:05 +0000 (UTC) Subject: [master] f4dffe593 Use "perfect hash" to find headers to be filtered out. Message-ID: <20210510180805.2BB1BAC2B5@lists.varnish-cache.org> commit f4dffe593b04a33f07423db3f9dc69eb428b2e85 Author: Poul-Henning Kamp Date: Thu Apr 29 11:02:21 2021 +0000 Use "perfect hash" to find headers to be filtered out. The actual agorithm was found with `gperf` but its output is not directly usable and include/tbl/http_headers.h change so infrequently that this step is not automated. (Asserts protect against overlooking this step if new headers are added to the table.) diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index 09fce7def..db0dad004 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -57,6 +57,131 @@ const char H__Status[] = "\010:status:"; const char H__Proto[] = "\007:proto:"; const char H__Reason[] = "\010:reason:"; +/*-------------------------------------------------------------------- + * Perfect hash to rapidly recognize headers from tbl/http_headers.h + * which have non-zero flags. + * + * A suitable algorithm can be found with `gperf`: + * + * tr '" ,' ' ' < include/tbl/http_headers.h | + * awk '$1 == "H(" && $4 != "0" {print$2}' | + * gperf --ignore-case + * + */ + +static const unsigned char http_asso_values[256] = { + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 25, 39, 0, 20, 5, 39, 39, 39, 15, 0, 39, + 10, 39, 0, 39, 15, 10, 39, 39, 0, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 25, 39, 0, 20, 5, 39, 39, 39, 15, 0, 39, + 10, 39, 0, 39, 15, 10, 39, 39, 0, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, + 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39 +}; + +static struct http_hdrflg { + char *hdr; + unsigned flag; +} http_hdrflg[38 + 1] = { // MAX_HASH_VALUE + { NULL }, + { NULL }, + { H_TE }, + { H_Age }, + { NULL }, + { H_Range }, + { NULL }, + { H_Upgrade }, + { H_If_Range }, + { NULL }, + { H_Connection }, + { NULL }, + { H_Trailer }, + { H_If_None_Match }, + { NULL }, + { NULL }, + { NULL }, + { H_Transfer_Encoding }, + { H_Proxy_Authenticate }, + { H_Proxy_Authorization }, + { H_Keep_Alive }, + { NULL }, + { NULL }, + { H_If_Match }, + { H_HTTP2_Settings }, + { NULL }, + { NULL }, + { NULL }, + { H_Content_Range }, + { H_If_Unmodified_Since }, + { NULL }, + { NULL }, + { H_If_Modified_Since }, + { H_Cache_Control }, + { NULL }, + { NULL }, + { NULL }, + { NULL }, + { H_Accept_Ranges } +}; + +static struct http_hdrflg * +http_hdr_flags(const char *b, const char *e) +{ + unsigned u; + struct http_hdrflg *retval; + + if (e == NULL) + return(NULL); + assert(e > b); + u = (unsigned)(e - b); + assert(b + u == e); + if (u < 2 || u > 19) // MIN_WORD_LENGTH & MAX_WORD_LENGTH + return(NULL); + if (u > 3) + u += http_asso_values[((const uint8_t*)b)[3]]; + if (u > 38) // MAX_HASH_VALUE + return(NULL); + retval = &http_hdrflg[u]; + if (retval->hdr == NULL) + return(NULL); + if (strncasecmp(retval->hdr + 1, b, e - b)) + return(NULL); + return(retval); +} + +/*--------------------------------------------------------------------*/ + +static void +http_init_hdr(char *hdr, int flg) +{ + struct http_hdrflg *f; + + hdr[0] = strlen(hdr + 1); + f = http_hdr_flags(hdr + 1, hdr + hdr[0]); + if (flg) { + AN(f); + assert(f->hdr == hdr); + f->flag = flg; + } +} + +void +HTTP_Init(void) +{ + +#define HTTPH(a, b, c) http_init_hdr(b, c); +#include "tbl/http_headers.h" +} + /*-------------------------------------------------------------------- * These two functions are in an incestuous relationship with the * order of macros in include/tbl/vsl_tags_http.h @@ -735,6 +860,7 @@ http_DoConnection(struct http *hp, enum sess_close sc_close) const char *h, *b, *e; enum sess_close retval; unsigned u, v; + struct http_hdrflg *f; CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC); assert(sc_close == SC_REQ_CLOSE || sc_close == SC_RESP_CLOSE); @@ -756,13 +882,9 @@ http_DoConnection(struct http *hp, enum sess_close sc_close) retval = SC_NULL; /* Refuse removal of well-known-headers if they would pass. */ -/*lint -save -e506 [constant value boolean] */ -#define HTTPH(a, x, c) \ - if (!((c) & HTTPH_R_PASS) && \ - strlen(a) == u && !strncasecmp(a, b, u)) \ + f = http_hdr_flags(b, e); + if (f != NULL && !(f->flag & HTTPH_R_PASS)) return (SC_RX_BAD); -#include "tbl/http_headers.h" -/*lint -restore */ for (v = HTTP_HDR_FIRST; v < hp->nhd; v++) { Tcheck(hp->hd[v]); @@ -898,13 +1020,16 @@ http_PutResponse(struct http *to, const char *proto, uint16_t status, static inline int http_isfiltered(const struct http *fm, unsigned u, unsigned how) { + const char *e; + const struct http_hdrflg *f; + if (fm->hdf[u] & HDF_FILTER) return (1); -#define HTTPH(a, b, c) \ - if (((c) & how) && http_IsHdr(&fm->hd[u], (b))) \ - return (1); -#include "tbl/http_headers.h" - return (0); + e = strchr(fm->hd[u].b, ':'); + if (e == NULL) + return (0); + f = http_hdr_flags(fm->hd[u].b, e); + return (f != NULL && f->flag & how); } int @@ -1313,13 +1438,3 @@ http_Unset(struct http *hp, const char *hdr) } hp->nhd = v; } - -/*--------------------------------------------------------------------*/ - -void -HTTP_Init(void) -{ - -#define HTTPH(a, b, c) b[0] = (char)strlen(b + 1); -#include "tbl/http_headers.h" -} diff --git a/bin/varnishtest/tests/c00016.vtc b/bin/varnishtest/tests/c00016.vtc index fa86a59bf..54316a627 100644 --- a/bin/varnishtest/tests/c00016.vtc +++ b/bin/varnishtest/tests/c00016.vtc @@ -1,10 +1,15 @@ -varnishtest "Test Connection header handling" +varnishtest "Test header filtering Table/Connection header" server s1 { rxreq expect req.url == "/foo" expect req.http.Foo == "bar" expect req.http.FromVCL == "123" + expect req.http.Proxy-Authenticate == "" + expect req.http.pROXY-aUTHENTICATE == "" + expect req.http.Proxy-Authenticat == "3" + expect req.http.Proxy-Authenticatd == "4" + expect req.http.Proxy-Authenticatef == "5" txresp -hdr "Bar: foo" -body "foobar" rxreq @@ -23,7 +28,13 @@ varnish v1 -vcl+backend { } -start client c1 { - txreq -url "/foo" -hdr "Foo: bar" + txreq -url "/foo" -hdr "Foo: bar" \ + -hdr "Proxy-Authenticate: 1" \ + -hdr "pROXY-aUTHENTICATE: 2" \ + -hdr "Proxy-Authenticat: 3" \ + -hdr "Proxy-Authenticatd: 4" \ + -hdr "Proxy-Authenticatef: 5" + rxresp expect resp.http.Bar == "foo" diff --git a/include/tbl/http_headers.h b/include/tbl/http_headers.h index 416d6a8f6..3fe1aaa00 100644 --- a/include/tbl/http_headers.h +++ b/include/tbl/http_headers.h @@ -36,6 +36,9 @@ * * see [RFC2616 13.5.1 End-to-end and Hop-by-hop Headers] * + * When fields with non-zero flags are added, the "perfect hash" at the + * top of cache_http.c will need to be reworked. See the comments there + * for instructions. */ /*lint -save -e525 -e539 */ From dridi.boukelmoune at gmail.com Tue May 11 05:57:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 11 May 2021 05:57:06 +0000 (UTC) Subject: [master] 3867a3984 travis: Drop distcheck from builds Message-ID: <20210511055706.9B7AA65759@lists.varnish-cache.org> commit 3867a3984d0bc343ad39a2243f1f3528af728ff5 Author: Dridi Boukelmoune Date: Tue May 11 07:52:14 2021 +0200 travis: Drop distcheck from builds There's no convenient way to parameterize configure arguments to pass them on to distcheck's configure step. We already run distcheck on circle ci and vtest, so it's simpler to simply run the check target. Refs 26e92d158080f34347d529b3952d0e6d4dd94f22 diff --git a/.travis.yml b/.travis.yml index 90ce54d2f..b3ac48bbf 100644 --- a/.travis.yml +++ b/.travis.yml @@ -26,12 +26,7 @@ jobs: - ./autogen.sh - ./configure --enable-maintainer-mode --with-unwind script: &script-common - - | - if [[ "$TRAVIS_PULL_REQUEST" != "false" ]]; then - make -j12 distcheck VERBOSE=1 - else - make -j16 check VERBOSE=1 - fi + - make -j16 check VERBOSE=1 - <<: *test-linux arch: arm64 - <<: *test-linux From dridi.boukelmoune at gmail.com Tue May 11 07:14:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 11 May 2021 07:14:05 +0000 (UTC) Subject: [master] eed91d391 travis: Disable developer warnings Message-ID: <20210511071405.606DA100B04@lists.varnish-cache.org> commit eed91d3910310e441c5acb44b1508cc10bdc5c47 Author: Dridi Boukelmoune Date: Tue May 11 09:08:24 2021 +0200 travis: Disable developer warnings They are currently triggering on readline/libedit includes, preventing the sanitizer job to fail for a good reason. We still have developer warnings as part of our circle and vtest ci. Better diff with the --word-diff option. diff --git a/.travis.yml b/.travis.yml index b3ac48bbf..af4c0b79a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -67,7 +67,7 @@ jobs: export UBSAN_OPTIONS=halt_on_error=1,print_stacktrace=1,use_sigaltstack=0,suppressions=$(pwd)/tools/ubsan.suppr export CC=clang-9 - ./autogen.sh - - ./configure --enable-maintainer-mode --with-unwind --enable-developer-warnings --enable-debugging-symbols --disable-stack-protector --with-persistent-storage --enable-asan --enable-ubsan + - ./configure --enable-maintainer-mode --with-unwind --enable-debugging-symbols --disable-stack-protector --with-persistent-storage --enable-asan --enable-ubsan - stage: test os: osx osx_image: xcode12 From phk at FreeBSD.org Tue May 11 11:12:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 11 May 2021 11:12:05 +0000 (UTC) Subject: [master] 69328819e Introduce a `hdr_t` typedef for our "pascal-string" H_* style args. Message-ID: <20210511111205.6975C106A95@lists.varnish-cache.org> commit 69328819e484a7e74a1cd5aa5e8f90c56e7480f3 Author: Poul-Henning Kamp Date: Tue May 11 11:11:08 2021 +0000 Introduce a `hdr_t` typedef for our "pascal-string" H_* style args. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 2ea190180..e770e23f1 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -69,6 +69,8 @@ struct body_status { typedef const struct body_status *body_status_t; +typedef const char *hdr_t; + /*--------------------------------------------------------------------*/ enum sess_close { @@ -594,40 +596,40 @@ void http_PutResponse(struct http *to, const char *proto, uint16_t status, void http_FilterReq(struct http *to, const struct http *fm, unsigned how); void HTTP_Encode(const struct http *fm, uint8_t *, unsigned len, unsigned how); int HTTP_Decode(struct http *to, const uint8_t *fm); -void http_ForceHeader(struct http *to, const char *hdr, const char *val); +void http_ForceHeader(struct http *to, hdr_t, const char *val); void http_PrintfHeader(struct http *to, const char *fmt, ...) v_printflike_(2, 3); void http_TimeHeader(struct http *to, const char *fmt, vtim_real now); void http_Proto(struct http *to); -void http_SetHeader(struct http *to, const char *hdr); -void http_SetH(struct http *to, unsigned n, const char *fm); +void http_SetHeader(struct http *to, const char *header); +void http_SetH(struct http *to, unsigned n, const char *header); void http_ForceField(struct http *to, unsigned n, const char *t); void HTTP_Setup(struct http *, struct ws *, struct vsl_log *, enum VSL_tag_e); void http_Teardown(struct http *ht); -int http_GetHdr(const struct http *hp, const char *hdr, const char **ptr); -int http_GetHdrToken(const struct http *hp, const char *hdr, +int http_GetHdr(const struct http *hp, hdr_t, const char **ptr); +int http_GetHdrToken(const struct http *hp, hdr_t, const char *token, const char **pb, const char **pe); -int http_GetHdrField(const struct http *hp, const char *hdr, +int http_GetHdrField(const struct http *hp, hdr_t, const char *field, const char **ptr); -double http_GetHdrQ(const struct http *hp, const char *hdr, const char *field); +double http_GetHdrQ(const struct http *hp, hdr_t, const char *field); ssize_t http_GetContentLength(const struct http *hp); uint16_t http_GetStatus(const struct http *hp); int http_IsStatus(const struct http *hp, int); void http_SetStatus(struct http *to, uint16_t status, const char *reason); const char *http_GetMethod(const struct http *hp); -int http_HdrIs(const struct http *hp, const char *hdr, const char *val); +int http_HdrIs(const struct http *hp, hdr_t, const char *val); void http_CopyHome(const struct http *hp); -void http_Unset(struct http *hp, const char *hdr); -unsigned http_CountHdr(const struct http *hp, const char *hdr); -void http_CollectHdr(struct http *hp, const char *hdr); -void http_CollectHdrSep(struct http *hp, const char *hdr, const char *sep); +void http_Unset(struct http *hp, hdr_t); +unsigned http_CountHdr(const struct http *hp, hdr_t); +void http_CollectHdr(struct http *hp, hdr_t); +void http_CollectHdrSep(struct http *hp, hdr_t, const char *sep); void http_VSL_log(const struct http *hp); void HTTP_Merge(struct worker *, struct objcore *, struct http *to); uint16_t HTTP_GetStatusPack(struct worker *, struct objcore *oc); int HTTP_IterHdrPack(struct worker *, struct objcore *, const char **); #define HTTP_FOREACH_PACK(wrk, oc, ptr) \ for ((ptr) = NULL; HTTP_IterHdrPack(wrk, oc, &(ptr));) -const char *HTTP_GetHdrPack(struct worker *, struct objcore *, const char *hdr); +const char *HTTP_GetHdrPack(struct worker *, struct objcore *, hdr_t); enum sess_close http_DoConnection(struct http *hp, enum sess_close sc_close); int http_IsFiltered(const struct http *hp, unsigned u, unsigned how); diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index db0dad004..ec0739ee6 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -390,12 +390,12 @@ http_Proto(struct http *to) /*--------------------------------------------------------------------*/ void -http_SetH(struct http *to, unsigned n, const char *fm) +http_SetH(struct http *to, unsigned n, const char *header) { assert(n < to->nhd); - AN(fm); - to->hd[n].b = TRUST_ME(fm); + AN(header); + to->hd[n].b = TRUST_ME(header); to->hd[n].e = strchr(to->hd[n].b, '\0'); to->hdf[n] = 0; http_VSLH(to, n); @@ -423,7 +423,7 @@ http_PutField(struct http *to, int field, const char *string) /*--------------------------------------------------------------------*/ static int -http_IsHdr(const txt *hh, const char *hdr) +http_IsHdr(const txt *hh, hdr_t hdr) { unsigned l; @@ -461,7 +461,7 @@ http_findhdr(const struct http *hp, unsigned l, const char *hdr) */ unsigned -http_CountHdr(const struct http *hp, const char *hdr) +http_CountHdr(const struct http *hp, hdr_t hdr) { unsigned retval = 0; unsigned u; @@ -482,7 +482,7 @@ http_CountHdr(const struct http *hp, const char *hdr) */ void -http_CollectHdr(struct http *hp, const char *hdr) +http_CollectHdr(struct http *hp, hdr_t hdr) { http_CollectHdrSep(hp, hdr, NULL); @@ -495,7 +495,7 @@ http_CollectHdr(struct http *hp, const char *hdr) */ void -http_CollectHdrSep(struct http *hp, const char *hdr, const char *sep) +http_CollectHdrSep(struct http *hp, hdr_t hdr, const char *sep) { unsigned u, l, lsep, ml, f, x, d; char *b = NULL, *e = NULL; @@ -578,7 +578,7 @@ http_CollectHdrSep(struct http *hp, const char *hdr, const char *sep) /*--------------------------------------------------------------------*/ int -http_GetHdr(const struct http *hp, const char *hdr, const char **ptr) +http_GetHdr(const struct http *hp, hdr_t hdr, const char **ptr) { unsigned u, l; const char *p; @@ -705,7 +705,7 @@ http_istoken(const char **bp, const char *e, const char *token) */ int -http_GetHdrToken(const struct http *hp, const char *hdr, +http_GetHdrToken(const struct http *hp, hdr_t hdr, const char *token, const char **pb, const char **pe) { const char *h, *b, *e; @@ -742,7 +742,7 @@ http_GetHdrToken(const struct http *hp, const char *hdr, */ double -http_GetHdrQ(const struct http *hp, const char *hdr, const char *field) +http_GetHdrQ(const struct http *hp, hdr_t hdr, const char *field) { const char *hb, *he, *b, *e; int i; @@ -791,7 +791,7 @@ http_GetHdrQ(const struct http *hp, const char *hdr, const char *field) */ int -http_GetHdrField(const struct http *hp, const char *hdr, +http_GetHdrField(const struct http *hp, hdr_t hdr, const char *field, const char **ptr) { const char *h; @@ -903,7 +903,7 @@ http_DoConnection(struct http *hp, enum sess_close sc_close) /*--------------------------------------------------------------------*/ int -http_HdrIs(const struct http *hp, const char *hdr, const char *val) +http_HdrIs(const struct http *hp, hdr_t hdr, const char *val) { const char *p; @@ -1183,7 +1183,7 @@ HTTP_IterHdrPack(struct worker *wrk, struct objcore *oc, const char **p) } const char * -HTTP_GetHdrPack(struct worker *wrk, struct objcore *oc, const char *hdr) +HTTP_GetHdrPack(struct worker *wrk, struct objcore *oc, hdr_t hdr) { const char *ptr; unsigned l; @@ -1343,22 +1343,22 @@ http_CopyHome(const struct http *hp) /*--------------------------------------------------------------------*/ void -http_SetHeader(struct http *to, const char *hdr) +http_SetHeader(struct http *to, const char *header) { CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); if (to->nhd >= to->shd) { - VSLb(to->vsl, SLT_LostHeader, "%s", hdr); + VSLb(to->vsl, SLT_LostHeader, "%s", header); http_fail(to); return; } - http_SetH(to, to->nhd++, hdr); + http_SetH(to, to->nhd++, header); } /*--------------------------------------------------------------------*/ void -http_ForceHeader(struct http *to, const char *hdr, const char *val) +http_ForceHeader(struct http *to, hdr_t hdr, const char *val) { CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); @@ -1420,7 +1420,7 @@ http_TimeHeader(struct http *to, const char *fmt, vtim_real now) /*--------------------------------------------------------------------*/ void -http_Unset(struct http *hp, const char *hdr) +http_Unset(struct http *hp, hdr_t hdr) { uint16_t u, v; From phk at FreeBSD.org Wed May 12 08:21:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 12 May 2021 08:21:06 +0000 (UTC) Subject: [master] c433abf47 Coverity spotted an unused assignment. Message-ID: <20210512082106.9EEEA103AE1@lists.varnish-cache.org> commit c433abf470f6298b42ddbfcc35d4f948c71a6a38 Author: Poul-Henning Kamp Date: Wed May 12 08:20:20 2021 +0000 Coverity spotted an unused assignment. diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 6b4825633..b063dee4c 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -439,7 +439,7 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk) tmo = now + 1.; else tmo = now + 60.; - i = Lck_CondWait(&wrk->cond, &pp->mtx, tmo); + (void)Lck_CondWait(&wrk->cond, &pp->mtx, tmo); if (wrk->task->func != NULL) { /* We have been handed a new task */ tpx = *wrk->task; From phk at FreeBSD.org Mon May 17 07:54:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 May 2021 07:54:05 +0000 (UTC) Subject: [master] 188dc97d6 Have backends send "Accept-ranges" headers where it matters. Message-ID: <20210517075405.20B121070DB@lists.varnish-cache.org> commit 188dc97d6ba818777de0a85911be1da57a276aa2 Author: Poul-Henning Kamp Date: Mon May 17 07:43:00 2021 +0000 Have backends send "Accept-ranges" headers where it matters. Prep work for #3251 diff --git a/bin/varnishtest/tests/l00002.vtc b/bin/varnishtest/tests/l00002.vtc index 7d6a03cf7..3ac66ece9 100644 --- a/bin/varnishtest/tests/l00002.vtc +++ b/bin/varnishtest/tests/l00002.vtc @@ -3,11 +3,11 @@ varnishtest "Test request byte counters" server s1 { rxreq expect req.url == "/1" - txresp -bodylen 1000 + txresp -hdr "Accept-ranges: bytes" -bodylen 1000 rxreq expect req.url == "/2" - txresp -bodylen 2000 + txresp -hdr "Accept-ranges: bytes" -bodylen 2000 } -start varnish v1 -vcl+backend { @@ -89,16 +89,20 @@ logexpect l1 -v v1 -g session { client c1 { txreq -method POST -url "/1" -hdr "Host: foo" -body "asdf" rxresp + expect resp.http.accept-ranges == "bytes" expect resp.status == 200 send "GET /2 HTTP/1.1\r\nHost: foo\r\n\r\nGET /2 HTTP/1.1\r\nHost: foo\r\n\r\n" rxresp + expect resp.http.accept-ranges == "bytes" expect resp.status == 200 rxresp + expect resp.http.accept-ranges == "bytes" expect resp.status == 200 send "GET\r\n\r\n" rxresp + expect resp.http.accept-ranges == "resp.http.accept-ranges" expect resp.status == 400 } -run diff --git a/bin/varnishtest/tests/t02005.vtc b/bin/varnishtest/tests/t02005.vtc index 03c9a85ae..6412f23c6 100644 --- a/bin/varnishtest/tests/t02005.vtc +++ b/bin/varnishtest/tests/t02005.vtc @@ -7,10 +7,10 @@ server s1 { rxreq expect req.http.content-length == 7 expect req.http.transfer-encoding == - txresp -hdr "Content-Type: text/plain" -body response + txresp -hdr "Accept-ranges: bytes" -hdr "Content-Type: text/plain" -body response rxreq - txresp + txresp -hdr "Accept-ranges: bytes" } -start varnish v1 -vcl+backend { From phk at FreeBSD.org Mon May 17 07:54:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 May 2021 07:54:05 +0000 (UTC) Subject: [master] b49127856 Dont force an Accept-Ranges header on non-cacheable responses. Message-ID: <20210517075405.3867D1070DE@lists.varnish-cache.org> commit b49127856b434369df69fa70fe26e0e49306695f Author: Poul-Henning Kamp Date: Mon May 17 07:52:15 2021 +0000 Dont force an Accept-Ranges header on non-cacheable responses. Fixes: #3251 diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 0d26f4615..34fde97a1 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -204,6 +204,7 @@ Resp_Setup_Synth(struct req *req) static enum req_fsm_nxt v_matchproto_(req_state_f) cnt_deliver(struct worker *wrk, struct req *req) { + unsigned status; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); @@ -223,6 +224,11 @@ cnt_deliver(struct worker *wrk, struct req *req) return (REQ_FSM_MORE); } + status = http_GetStatus(req->resp); + if (cache_param->http_range_support && status == 200 && + !(req->objcore->flags & OC_F_PRIVATE)) + http_ForceHeader(req->resp, H_Accept_Ranges, "bytes"); + VCL_deliver_method(req->vcl, wrk, req, NULL, NULL); VSLb_ts_req(req, "Process", W_TIM_real(wrk)); @@ -437,9 +443,6 @@ cnt_transmit(struct worker *wrk, struct req *req) VSLb(req->vsl, SLT_Error, "Failure to push processors"); req->doclose = SC_OVERLOAD; } else { - if (cache_param->http_range_support && status == 200) - http_ForceHeader(req->resp, H_Accept_Ranges, "bytes"); - if (status < 200 || status == 204) { // rfc7230,l,1691,1695 http_Unset(req->resp, H_Content_Length); @@ -465,8 +468,8 @@ cnt_transmit(struct worker *wrk, struct req *req) } if (req->resp_len == 0) sendbody = 0; + req->transport->deliver(req, boc, sendbody); } - req->transport->deliver(req, boc, sendbody); VSLb_ts_req(req, "Resp", W_TIM_real(wrk)); diff --git a/bin/varnishtest/tests/c00034.vtc b/bin/varnishtest/tests/c00034.vtc index d5432a430..c3f904663 100644 --- a/bin/varnishtest/tests/c00034.vtc +++ b/bin/varnishtest/tests/c00034.vtc @@ -16,6 +16,7 @@ varnish v1 -cliok "param.set http_range_support off" client c1 { txreq -hdr "Range: bytes=0-9" rxresp + expect resp.accept-ranges == "resp.accept-ranges" expect resp.status == 200 expect resp.bodylen == 100 } -run @@ -85,6 +86,7 @@ client c1 { expect resp.status == 206 expect resp.bodylen == 50 expect resp.http.content-range == "bytes 0-49/100" + expect resp.http.accept-ranges == "bytes" txreq -hdr "Range: bytes=50-99" rxresp @@ -211,3 +213,32 @@ client c2 { expect resp.body == BL } -run } -run + +varnish v1 -vsl_catchup + +server s1 { + rxreq + txresp + rxreq + txresp -hdr "Accept-Ranges: foobar" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + return (pass); + } + sub vcl_deliver { + set resp.http.foobar = resp.http.accept-ranges; + } +} + +client c1 { + txreq + rxresp + expect resp.http.foobar == "" + expect resp.http.accept-ranges == resp.http.accept-ranges + txreq + rxresp + expect resp.http.foobar == "foobar" + expect resp.http.accept-ranges == foobar +} -run diff --git a/doc/changes.rst b/doc/changes.rst index b903369cd..aa6e744d6 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -35,6 +35,9 @@ release process. Varnish Cache 7.x.x (2021-09-15) ================================ +* Accept-Ranges headers are no longer generated for passed objects, + but must either come from the backend or be created in `vcl_deliver{}` + * ACLs no longer produce VSL `VCL_acl` records by default, this must be explicitly enabled with `vcl +log { ... }`. From phk at FreeBSD.org Mon May 17 09:05:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 May 2021 09:05:06 +0000 (UTC) Subject: [master] 3dd4db765 Check obj.uncacheable says the right thing. Message-ID: <20210517090506.E5C0C108FE9@lists.varnish-cache.org> commit 3dd4db7655cc5a4105f1f8ea2cc2e25da9e43728 Author: Poul-Henning Kamp Date: Mon May 17 09:03:50 2021 +0000 Check obj.uncacheable says the right thing. Spotted by: @dridi diff --git a/bin/varnishtest/tests/c00011.vtc b/bin/varnishtest/tests/c00011.vtc index 907355f2f..b183f0dc2 100644 --- a/bin/varnishtest/tests/c00011.vtc +++ b/bin/varnishtest/tests/c00011.vtc @@ -38,6 +38,7 @@ client c1 { expect resp.http.o_ttl <= 120 expect resp.http.o_grace == "10.000" expect resp.http.o_keep == "0.000" + expect resp.http.o_uncacheable == "true" txreq -url "/foo" rxresp @@ -50,6 +51,7 @@ client c1 { expect resp.http.o_ttl <= 120 expect resp.http.o_grace == "10.000" expect resp.http.o_keep == "0.000" + expect resp.http.o_uncacheable == "true" } client c1 -run diff --git a/bin/varnishtest/tests/c00012.vtc b/bin/varnishtest/tests/c00012.vtc index 9f64f6883..67c6b2a66 100644 --- a/bin/varnishtest/tests/c00012.vtc +++ b/bin/varnishtest/tests/c00012.vtc @@ -34,6 +34,7 @@ client c1 { expect resp.http.o_ttl > -0.5 expect resp.http.o_grace == "0.000" expect resp.http.o_keep == "0.000" + expect resp.http.o_uncacheable == "true" txreq -url "/foo" rxresp @@ -46,6 +47,7 @@ client c1 { expect resp.http.o_ttl > -0.5 expect resp.http.o_grace == "0.000" expect resp.http.o_keep == "0.000" + expect resp.http.o_uncacheable == "true" } client c1 -run From phk at FreeBSD.org Mon May 17 11:31:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 May 2021 11:31:05 +0000 (UTC) Subject: [master] 1adbf27d7 Check obj.uncacheable Message-ID: <20210517113105.2163410EEA9@lists.varnish-cache.org> commit 1adbf27d7ae73e02509010eefb5ddff9abb6d697 Author: Poul-Henning Kamp Date: Mon May 17 11:23:21 2021 +0000 Check obj.uncacheable diff --git a/bin/varnishtest/tests/b00002.vtc b/bin/varnishtest/tests/b00002.vtc index 2c978ee54..ade1169d3 100644 --- a/bin/varnishtest/tests/b00002.vtc +++ b/bin/varnishtest/tests/b00002.vtc @@ -13,6 +13,9 @@ varnish v1 -arg "-sTransient=default,1m" -vcl+backend { sub vcl_backend_response { set beresp.http.x-ttl = beresp.ttl; } + sub vcl_deliver { + set resp.http.o_uncacheable = obj.uncacheable; + } } -start # check that there are no TTL LogTags between the @@ -32,11 +35,9 @@ client c1 { expect resp.status == 200 expect resp.http.connection == close expect resp.http.x-ttl == 0.000 + expect resp.http.o_uncacheable == true } -run -# Give varnish a chance to update stats -delay .1 - varnish v1 -expect n_object == 0 varnish v1 -expect SM?.Transient.g_alloc == 0 varnish v1 -expect sess_conn == 1 From phk at FreeBSD.org Mon May 17 11:31:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 May 2021 11:31:05 +0000 (UTC) Subject: [master] 9aa7fae23 Busyobj->do_pass was surplus to requirements, ->uncacheable is plenty. Message-ID: <20210517113105.37E1810EEAC@lists.varnish-cache.org> commit 9aa7fae23f6f2eeb1831be2383d8eb688d892f28 Author: Poul-Henning Kamp Date: Mon May 17 11:29:06 2021 +0000 Busyobj->do_pass was surplus to requirements, ->uncacheable is plenty. diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 4bb321b78..273eeb16b 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -259,9 +259,9 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo) HTTP_Setup(bo->bereq0, bo->ws, bo->vsl, SLT_BereqMethod); http_FilterReq(bo->bereq0, bo->req->http, - bo->do_pass ? HTTPH_R_PASS : HTTPH_R_FETCH); + bo->uncacheable ? HTTPH_R_PASS : HTTPH_R_FETCH); - if (bo->do_pass) + if (bo->uncacheable) AZ(bo->stale_oc); else { http_ForceField(bo->bereq0, HTTP_HDR_METHOD, "GET"); @@ -366,7 +366,7 @@ vbf_304_logic(struct busyobj *bo) HTTP_Merge(bo->wrk, bo->stale_oc, bo->beresp); assert(http_IsStatus(bo->beresp, 200)); bo->was_304 = 1; - } else if (!bo->do_pass) { + } else if (!bo->uncacheable) { /* * Backend sent unallowed 304 */ @@ -397,7 +397,7 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC); AZ(bo->storage); - bo->storage = bo->do_pass ? stv_transient : STV_next(); + bo->storage = bo->uncacheable ? stv_transient : STV_next(); if (bo->retries > 0) http_Unset(bo->bereq, "\012X-Varnish:"); @@ -406,7 +406,6 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) VCL_backend_fetch_method(bo->vcl, wrk, NULL, bo, NULL); - bo->uncacheable = bo->do_pass; if (wrk->handling == VCL_RET_ABANDON || wrk->handling == VCL_RET_FAIL) return (F_STP_FAIL); @@ -538,7 +537,7 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) bo->uncacheable = 1; wrk->handling = VCL_RET_DELIVER; } - if (bo->do_pass || bo->uncacheable) + if (bo->uncacheable) oc->flags |= OC_F_HFM; assert(wrk->handling == VCL_RET_DELIVER); @@ -1114,7 +1113,7 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc, switch (mode) { case VBF_PASS: how = "pass"; - bo->do_pass = 1; + bo->uncacheable = 1; break; case VBF_NORMAL: how = "fetch"; diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 34fde97a1..b39fa6fe5 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -201,6 +201,8 @@ Resp_Setup_Synth(struct req *req) http_SetHeader(h, "Connection: close"); } +#include + static enum req_fsm_nxt v_matchproto_(req_state_f) cnt_deliver(struct worker *wrk, struct req *req) { diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index 4a1e50860..186de5092 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -262,7 +262,7 @@ VRT_r_bereq_uncacheable(VRT_CTX) { CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC); - return (ctx->bo->do_pass); + return (ctx->bo->uncacheable); } VCL_VOID diff --git a/include/tbl/bo_flags.h b/include/tbl/bo_flags.h index ac0fdf241..4a4f1dd4b 100644 --- a/include/tbl/bo_flags.h +++ b/include/tbl/bo_flags.h @@ -39,7 +39,6 @@ BO_FLAG(do_esi, 0, 1, 1, 1, "") BO_FLAG(do_gzip, 0, 1, 1, 1, "") BO_FLAG(do_gunzip, 0, 1, 1, 1, "") BO_FLAG(do_stream, 0, 1, 1, 0, "") -BO_FLAG(do_pass, 0, 0, 0, 0, "") BO_FLAG(uncacheable, 0, 0, 0, 0, "") BO_FLAG(was_304, 0, 1, 0, 0, "") BO_FLAG(is_bgfetch, 1, 0, 0, 0, "") From dridi.boukelmoune at gmail.com Mon May 17 12:00:08 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 17 May 2021 12:00:08 +0000 (UTC) Subject: [master] 776b67400 objcore: Rename the ABANDON flag to CANCEL Message-ID: <20210517120008.318A310FC97@lists.varnish-cache.org> commit 776b67400ebd97a3ea29466e2354853a0098d61a Author: Dridi Boukelmoune Date: Mon May 17 13:57:29 2021 +0200 objcore: Rename the ABANDON flag to CANCEL This is to avoid a confusion between cancelling a fetch from a client transaction and abandoning a fetch from a backend transaction. diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 273eeb16b..8c641e5e4 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -571,7 +571,7 @@ vbf_stp_fetchbody(struct worker *wrk, struct busyobj *bo) est = 0; do { - if (oc->flags & OC_F_ABANDON) { + if (oc->flags & OC_F_CANCEL) { /* * A pass object and delivery was terminated * We don't fail the fetch, in order for HitMiss diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index 2855c71e9..d9a4e0c69 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -763,11 +763,11 @@ HSH_Fail(struct objcore *oc) } /*--------------------------------------------------------------------- - * Abandon a fetch we will not need + * Mark a fetch we will not need as cancelled */ static void -hsh_abandon(struct objcore *oc) +hsh_cancel(struct objcore *oc) { struct objhead *oh; @@ -776,7 +776,7 @@ hsh_abandon(struct objcore *oc) CHECK_OBJ(oh, OBJHEAD_MAGIC); Lck_Lock(&oh->mtx); - oc->flags |= OC_F_ABANDON; + oc->flags |= OC_F_CANCEL; Lck_Unlock(&oh->mtx); } @@ -807,7 +807,7 @@ HSH_Cancel(struct worker *wrk, struct objcore *oc, struct boc *boc) AN(oc->flags & OC_F_HFM); if (boc != NULL) { - hsh_abandon(oc); + hsh_cancel(oc); ObjWaitState(oc, BOS_FINISHED); } diff --git a/include/tbl/oc_flags.h b/include/tbl/oc_flags.h index 6bde9d19e..9a37335f2 100644 --- a/include/tbl/oc_flags.h +++ b/include/tbl/oc_flags.h @@ -34,7 +34,7 @@ OC_FLAG(PURGED, purged, (1<<0)) //lint !e835 OC_FLAG(BUSY, busy, (1<<1)) OC_FLAG(HFM, hfm, (1<<2)) OC_FLAG(HFP, hfp, (1<<3)) -OC_FLAG(ABANDON, abandon, (1<<4)) +OC_FLAG(CANCEL, cancel, (1<<4)) OC_FLAG(PRIVATE, private, (1<<5)) OC_FLAG(FAILED, failed, (1<<6)) OC_FLAG(DYING, dying, (1<<7)) From dridi.boukelmoune at gmail.com Tue May 18 05:08:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 18 May 2021 05:08:06 +0000 (UTC) Subject: [master] d59106073 Remove leftover include Message-ID: <20210518050806.852D2104C70@lists.varnish-cache.org> commit d591060732c64034c4ff9dc8498dc86d61090f23 Author: Dridi Boukelmoune Date: Tue May 18 07:04:43 2021 +0200 Remove leftover include Refs 9aa7fae23f6f2eeb1831be2383d8eb688d892f28 diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index b39fa6fe5..34fde97a1 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -201,8 +201,6 @@ Resp_Setup_Synth(struct req *req) http_SetHeader(h, "Connection: close"); } -#include - static enum req_fsm_nxt v_matchproto_(req_state_f) cnt_deliver(struct worker *wrk, struct req *req) { From phk at FreeBSD.org Wed May 19 07:41:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 07:41:06 +0000 (UTC) Subject: [master] 4656a5225 Dont use "scientific notation" for parameters. Message-ID: <20210519074106.68B72108A20@lists.varnish-cache.org> commit 4656a5225876ae6dc3133b4afb56bd0d2b0e630b Author: Poul-Henning Kamp Date: Wed May 19 06:54:12 2021 +0000 Dont use "scientific notation" for parameters. diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index b063dee4c..32b53024d 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -685,7 +685,7 @@ pool_herder(void *priv) if (pp->die) { if (delay < 2) - delay = 10e-3; + delay = .01; else delay = 1; VTIM_sleep(delay); diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 939b2a150..721e05292 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -434,7 +434,7 @@ tweak_poolparam(struct vsb *vsb, const struct parspec *par, const char *arg) if (retval) break; retval = tweak_generic_double(vsb, - &px.max_age, av[3], "0", "1e6", "%.0f"); + &px.max_age, av[3], "0", "1000000", "%.0f"); if (retval) break; if (px.min_pool > px.max_pool) { diff --git a/include/tbl/params.h b/include/tbl/params.h index 763920984..5d1a3952e 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -1424,7 +1424,7 @@ PARAM_THREAD( /* name */ thread_pool_fail_delay, /* field */ fail_delay, /* type */ timeout, - /* min */ "10e-3", + /* min */ ".01", /* max */ NULL, /* def */ "0.2", /* units */ "seconds", From phk at FreeBSD.org Wed May 19 07:41:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 07:41:06 +0000 (UTC) Subject: [master] 1824a3482 Discontinue use of "scientific notation" in test-cases. Message-ID: <20210519074106.7DAFE108A23@lists.varnish-cache.org> commit 1824a3482fd356d0f639cfc0c8ee72167cfc2dd8 Author: Poul-Henning Kamp Date: Wed May 19 07:02:01 2021 +0000 Discontinue use of "scientific notation" in test-cases. diff --git a/bin/varnishtest/tests/r03308.vtc b/bin/varnishtest/tests/r03308.vtc index 6c60821c9..144dcf8e9 100644 --- a/bin/varnishtest/tests/r03308.vtc +++ b/bin/varnishtest/tests/r03308.vtc @@ -11,7 +11,7 @@ varnish v1 -vcl+backend { import std; sub vcl_deliver { - set resp.http.ts = std.real2time(std.real("1e+22", 0), now); + set resp.http.ts = std.real2time(std.real("10000000000000000000000", 0), now); } } -start diff --git a/bin/varnishtest/tests/r03463.vtc b/bin/varnishtest/tests/r03463.vtc index 9d0818e7f..3d14ea9d0 100644 --- a/bin/varnishtest/tests/r03463.vtc +++ b/bin/varnishtest/tests/r03463.vtc @@ -8,7 +8,7 @@ varnish v1 -vcl { std.log("float1: 123.456"); std.log("float2: 123."); std.log("float3: .456"); - std.log("float4: 12.3e1"); + std.log("float4: 123"); std.log("float5: e1"); } return (synth(200)); From phk at FreeBSD.org Wed May 19 07:41:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 07:41:06 +0000 (UTC) Subject: [master] d127ca6e6 Discontinue support for 'scientific notation'. Message-ID: <20210519074106.9C95C108A26@lists.varnish-cache.org> commit d127ca6e63da6e58adca569466b2b56e00b6d528 Author: Poul-Henning Kamp Date: Wed May 19 07:37:41 2021 +0000 Discontinue support for 'scientific notation'. Considerable work went into RFC8941 to establish that there is neither a nead nor a reason to support parsing scientific notation in HTTP header context. diff --git a/doc/changes.rst b/doc/changes.rst index aa6e744d6..92f3c533e 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -35,6 +35,11 @@ release process. Varnish Cache 7.x.x (2021-09-15) ================================ +* 'Scientific Notation' numbers like 6.62607004e-34 are no longer + supported in VCL. (The preparation of RFC8941 made it clear that + there are neither reason nor any need to support scientific notation + in context of HTTP headers. + * Accept-Ranges headers are no longer generated for passed objects, but must either come from the backend or be created in `vcl_deliver{}` diff --git a/lib/libvarnish/vnum.c b/lib/libvarnish/vnum.c index 4e0d004f0..84b2acff8 100644 --- a/lib/libvarnish/vnum.c +++ b/lib/libvarnish/vnum.c @@ -83,15 +83,6 @@ VNUMpfx(const char *p, const char **t) } if (e > 0.0) return (nan("")); // No digits - if (*p == 'e' || *p == 'E') { - p++; - if (*p == '-' || *p == '+') - es = (*p++ == '-' ? -1.0 : 1.0); - if (!vct_isdigit(*p)) - return (nan("")); - for (; vct_isdigit(*p); p++) - ee = ee * 10. + *p - '0'; - } while (vct_issp(*p)) p++; if (*p != '\0') @@ -245,7 +236,10 @@ VNUM_2bytes(const char *p, uintmax_t *r, uintmax_t rel) } #ifdef NUM_C_TEST -/* Compile with: "cc -o foo -DNUM_C_TEST -I../.. -I../../include num.c -lm" */ +/* + * Compile with: + * cc -o foo -DNUM_C_TEST -I../.. -I../../include vnum.c vas.c vct.c -lm + */ static struct test_case { const char *str; @@ -315,11 +309,11 @@ static const char *vec[] = { " 12.", " 12.3", " 12.34", - " 12.34e-3", - " 12.34e3", - " 12.34e+3", - " +12.34e-3", - " -12.34e3", + "N12.34e-3", + "N12.34e3", + "N12.34e+3", + "N+12.34e-3", + "N-12.34e3", "N.", "N.12.", "N12..", From phk at FreeBSD.org Wed May 19 09:09:04 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 09:09:04 +0000 (UTC) Subject: [master] d1a8c7bc0 Restrict ttls to valid RFC8941 sf-decimal values Message-ID: <20210519090905.05E9110D42D@lists.varnish-cache.org> commit d1a8c7bc02ff7c8712c434d4b5c3d51c9ba9a061 Author: Poul-Henning Kamp Date: Wed May 19 09:06:25 2021 +0000 Restrict ttls to valid RFC8941 sf-decimal values diff --git a/bin/varnishtest/tests/r01499.vtc b/bin/varnishtest/tests/r01499.vtc index 4f313e5da..9c61f90ad 100644 --- a/bin/varnishtest/tests/r01499.vtc +++ b/bin/varnishtest/tests/r01499.vtc @@ -10,7 +10,7 @@ server s1 { varnish v1 -vcl+backend { sub vcl_backend_response { - set beresp.ttl = 0.0001s; + set beresp.ttl = 0.001s; set beresp.grace = 0s; set beresp.keep = 2s; } diff --git a/bin/varnishtest/tests/r01879.vtc b/bin/varnishtest/tests/r01879.vtc index 8c3d7e296..090adc601 100644 --- a/bin/varnishtest/tests/r01879.vtc +++ b/bin/varnishtest/tests/r01879.vtc @@ -17,7 +17,7 @@ varnish v1 -vcl+backend { import std; sub vcl_backend_response { - set beresp.ttl = 0.00001s; + set beresp.ttl = 0.001s; set beresp.grace = 0.1s; set beresp.keep = 9999s; } diff --git a/bin/varnishtest/tests/r02319.vtc b/bin/varnishtest/tests/r02319.vtc index 005ab0fe0..0ae5e41d1 100644 --- a/bin/varnishtest/tests/r02319.vtc +++ b/bin/varnishtest/tests/r02319.vtc @@ -8,7 +8,7 @@ server s1 { varnish v1 -vcl+backend { sub vcl_backend_fetch { - set bereq.between_bytes_timeout = 0.0001s; + set bereq.between_bytes_timeout = 0.001s; } sub vcl_backend_response { set beresp.do_stream = false; diff --git a/bin/varnishtest/tests/r02763.vtc b/bin/varnishtest/tests/r02763.vtc index 3488d7c95..8babdadf7 100644 --- a/bin/varnishtest/tests/r02763.vtc +++ b/bin/varnishtest/tests/r02763.vtc @@ -29,7 +29,7 @@ varnish v1 -vcl+backend { if (bereq.http.HFM) { set beresp.uncacheable = true; } else { - set beresp.ttl = 0.0001s; + set beresp.ttl = 0.001s; set beresp.grace = 0s; set beresp.keep = 1m; } diff --git a/bin/varnishtest/tests/r02946.vtc b/bin/varnishtest/tests/r02946.vtc index 71872679a..3756b9845 100644 --- a/bin/varnishtest/tests/r02946.vtc +++ b/bin/varnishtest/tests/r02946.vtc @@ -12,7 +12,7 @@ varnish v1 -vcl { return (deliver); } set beresp.status = 200; - set beresp.ttl = 0.0001s; + set beresp.ttl = 0.001s; set beresp.grace = 1h; return (deliver); } From phk at FreeBSD.org Wed May 19 12:29:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 12:29:06 +0000 (UTC) Subject: [master] 1beb644ec Add VRT support for VCL_INTEGER and VCL_REAL validity Message-ID: <20210519122906.BF6F6112615@lists.varnish-cache.org> commit 1beb644ec4356ff87281d03b3d583130816a1695 Author: Poul-Henning Kamp Date: Wed May 19 10:58:11 2021 +0000 Add VRT support for VCL_INTEGER and VCL_REAL validity diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c index 59ebaf697..82cd54e53 100644 --- a/bin/varnishd/cache/cache_vrt.c +++ b/bin/varnishd/cache/cache_vrt.c @@ -817,6 +817,13 @@ VRT_IP_string(VRT_CTX, VCL_IP ip) return (p); } +int +VRT_INT_is_valid(VCL_INT arg) +{ + return (arg >= VRT_INTEGER_MIN && arg <= VRT_INTEGER_MAX); +} + + VCL_STRING v_matchproto_() VRT_INT_string(VRT_CTX, VCL_INT num) { @@ -825,6 +832,12 @@ VRT_INT_string(VRT_CTX, VCL_INT num) return (WS_Printf(ctx->ws, "%jd", (intmax_t)num)); } +int +VRT_REAL_is_valid(VCL_REAL arg) +{ + return (!isnan(arg) && arg >= VRT_DECIMAL_MIN && arg <= VRT_DECIMAL_MAX); +} + VCL_STRING v_matchproto_() VRT_REAL_string(VRT_CTX, VCL_REAL num) { diff --git a/include/vrt.h b/include/vrt.h index 10528a3e5..31198fc95 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -301,6 +301,16 @@ typedef vtim_real VCL_TIME; typedef struct vcl * VCL_VCL; typedef void VCL_VOID; +/* + * These limits track RFC8941 + * We use hex notation because 999999999999.999 is not perfectly + * representable in ieee64 doubles. + */ +#define VRT_INTEGER_MAX 999999999999999 +#define VRT_INTEGER_MIN -999999999999999 +#define VRT_DECIMAL_MAX 0x1.d1a94a1fffff8p+39 +#define VRT_DECIMAL_MIN -0x1.d1a94a1fffff8p+39 + /*********************************************************************** * This is the composite "context" argument for compiled VCL, VRT and * VMOD functions. @@ -412,11 +422,17 @@ VCL_BACKEND VRT_DirectorResolve(VRT_CTX, VCL_BACKEND); /* VCL_BLOB */ VCL_BLOB VRT_blob(VRT_CTX, const char *, const void *, size_t, unsigned); +/* VCL_INT */ +int VRT_INT_is_valid(VCL_INT); + /* VCL_IP */ int VRT_VSA_GetPtr(VRT_CTX, VCL_IP sua, const unsigned char ** dst); VCL_BOOL VRT_ipcmp(VRT_CTX, VCL_IP, VCL_IP); void VRT_Format_Proxy(struct vsb *, VCL_INT, VCL_IP, VCL_IP, VCL_STRING); +/* VCL_REAL */ +int VRT_REAL_is_valid(VCL_REAL); + /* VCL_REGEX */ VCL_BOOL VRT_re_match(VRT_CTX, VCL_STRING, VCL_REGEX); VCL_STRING VRT_regsub(VRT_CTX, int all, VCL_STRING, VCL_REGEX, VCL_STRING); From phk at FreeBSD.org Wed May 19 12:29:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 12:29:06 +0000 (UTC) Subject: [master] 81822cc82 Dont render VCL_BYTES with millibytes precision Message-ID: <20210519122906.D8028112618@lists.varnish-cache.org> commit 81822cc8239732f4ab6950678feb1474c49b13eb Author: Poul-Henning Kamp Date: Wed May 19 11:49:13 2021 +0000 Dont render VCL_BYTES with millibytes precision diff --git a/bin/varnishtest/tests/m00005.vtc b/bin/varnishtest/tests/m00005.vtc index 253f49584..970e26e8e 100644 --- a/bin/varnishtest/tests/m00005.vtc +++ b/bin/varnishtest/tests/m00005.vtc @@ -46,42 +46,42 @@ client c1 { txreq -hdr "duration: 0.010s" rxresp expect resp.http.duration == 1000000.010 - expect resp.http.bytes == 10752.000 + expect resp.http.bytes == 10752 txreq -hdr "duration: 10ms" -hdr "bytes: 2k" rxresp expect resp.http.duration == 1000000.010 - expect resp.http.bytes == 2560.000 + expect resp.http.bytes == 2560 txreq -hdr "duration: 10.1s" -hdr "bytes: 3 m" rxresp expect resp.http.duration == 1000010.100 - expect resp.http.bytes == 3146240.000 + expect resp.http.bytes == 3146240 txreq -hdr "duration: 10m" -hdr "bytes:4.5 g" rxresp expect resp.http.duration == 1000600.000 - expect resp.http.bytes == 4831838720.000 + expect resp.http.bytes == 4831838720 txreq -hdr "duration: 10h" -hdr "bytes: 0.12 TB" rxresp expect resp.http.duration == 1036000.000 - expect resp.http.bytes == 131941395845.000 + expect resp.http.bytes == 131941395845 txreq -hdr "duration: 10d" -hdr "bytes: 0.25 PB" rxresp expect resp.http.duration == 1864000.000 - expect resp.http.bytes == 281474976711168.000 + expect resp.http.bytes == 281474976711168 txreq -hdr "duration: 10w" -hdr "bytes: 34%" rxresp expect resp.http.duration == 7048000.000 - expect resp.http.bytes == 10752.000 + expect resp.http.bytes == 10752 txreq -hdr "duration: 1y" -hdr "bytes: 34x" rxresp expect resp.http.duration == 32536000.000 - expect resp.http.bytes == 10752.000 + expect resp.http.bytes == 10752 txreq -hdr "duration: -100s" rxresp @@ -116,22 +116,22 @@ client c1 { txreq -hdr "d-real: 0.010" -hdr "b-real: 2048.99999" rxresp expect resp.http.duration == 1000000.010 - expect resp.http.bytes == 2560.000 + expect resp.http.bytes == 2560 txreq -hdr "d-real: 10.1" -hdr "b-real: 3145728.9" rxresp expect resp.http.duration == 1000010.100 - expect resp.http.bytes == 3146240.000 + expect resp.http.bytes == 3146240 txreq -hdr "d-real: 600" -hdr "b-real: 4831838208.123" rxresp expect resp.http.duration == 1000600.000 - expect resp.http.bytes == 4831838720.000 + expect resp.http.bytes == 4831838720 txreq -hdr "d-real: 36000" -hdr "b-real: 131941395333.12" rxresp expect resp.http.duration == 1036000.000 - expect resp.http.bytes == 131941395845.000 + expect resp.http.bytes == 131941395845 txreq -hdr "d-real: -100" rxresp @@ -146,12 +146,12 @@ client c1 { txreq -hdr "d-int: 600" -hdr "b-int: 4831838208" rxresp expect resp.http.duration == 1000600.000 - expect resp.http.bytes == 4831838720.000 + expect resp.http.bytes == 4831838720 txreq -hdr "d-int: 36000" -hdr "b-int: 131941395333" rxresp expect resp.http.duration == 1036000.000 - expect resp.http.bytes == 131941395845.000 + expect resp.http.bytes == 131941395845 txreq -hdr "d-int: -100" rxresp diff --git a/bin/varnishtest/tests/r03131.vtc b/bin/varnishtest/tests/r03131.vtc index c70f6500f..ad6fc0e95 100644 --- a/bin/varnishtest/tests/r03131.vtc +++ b/bin/varnishtest/tests/r03131.vtc @@ -21,14 +21,14 @@ varnish v1 -vcl { } -start logexpect l1 -v v1 -g raw { - expect * * VCL_Log {^\Qres(8) = 8.000\E$} - expect 0 = VCL_Log {^\Qres(15) = 16.000\E$} - expect 0 = VCL_Log {^\Qres(16) = 16.000\E$} + expect * * VCL_Log {^\Qres(8) = 8\E$} + expect 0 = VCL_Log {^\Qres(15) = 16\E$} + expect 0 = VCL_Log {^\Qres(16) = 16\E$} - expect 0 = VCL_Log {^\Qres(17) = 0.000\E$} + expect 0 = VCL_Log {^\Qres(17) = 0\E$} # workspace is now overflown, but smaller reservation still succeeds - expect 0 = VCL_Log {^\Qres(8) = 8.000\E$} + expect 0 = VCL_Log {^\Qres(8) = 8\E$} expect * = Error {^workspace_session overflow$} } -start diff --git a/bin/varnishtest/tests/v00033.vtc b/bin/varnishtest/tests/v00033.vtc index fcd7e422b..5d9b1e6b4 100644 --- a/bin/varnishtest/tests/v00033.vtc +++ b/bin/varnishtest/tests/v00033.vtc @@ -27,13 +27,13 @@ client c1 { txreq rxresp expect resp.status == 200 - expect resp.http.foo == 1100586419201.000 + expect resp.http.foo == 1100586419201 expect resp.http.bar == false txreq -url /foo rxresp expect resp.status == 200 - expect resp.http.foo == 1100586419201.000 + expect resp.http.foo == 1100586419201 expect resp.http.bar == true } -run diff --git a/lib/libvcc/vcc_types.c b/lib/libvcc/vcc_types.c index 7fcf5078c..fc3c01cd6 100644 --- a/lib/libvcc/vcc_types.c +++ b/lib/libvcc/vcc_types.c @@ -93,7 +93,7 @@ const struct type BOOL[1] = {{ const struct type BYTES[1] = {{ .magic = TYPE_MAGIC, .name = "BYTES", - .tostring = "VRT_REAL_string(ctx, \v1)", // XXX: wrong + .tostring = "VRT_INT_string(ctx, \v1)", .multype = REAL, // XXX: wrong }}; From phk at FreeBSD.org Wed May 19 12:29:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 12:29:06 +0000 (UTC) Subject: [master] 4502a1b9d VCL_BYTES is a subtype of VCL_INT Message-ID: <20210519122907.0163F11261C@lists.varnish-cache.org> commit 4502a1b9d21d8fd3e495ed85d640e6c425b8dc4f Author: Poul-Henning Kamp Date: Wed May 19 11:55:52 2021 +0000 VCL_BYTES is a subtype of VCL_INT diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 58308d951..5c4139153 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -431,7 +431,7 @@ void Resolve_Sockaddr(struct vcc *tl, const char *host, const char *defport, const char **ipv6_ascii, const char **p_ascii, int maxips, const struct token *t_err, const char *errid); double vcc_DurationUnit(struct vcc *); -void vcc_ByteVal(struct vcc *, double *); +void vcc_ByteVal(struct vcc *, VCL_INT *); void vcc_Duration(struct vcc *tl, double *); unsigned vcc_UintVal(struct vcc *tl); int vcc_IsFlag(struct vcc *tl); diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 3a762a1ee..99f42df66 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -689,7 +689,7 @@ vcc_expr5(struct vcc *tl, struct expr **e, vcc_type_t fmt) const char *ip, *sign; struct token *t; struct symbol *sym; - double d; + VCL_INT vi; sign = ""; *e = NULL; @@ -805,9 +805,9 @@ vcc_expr5(struct vcc *tl, struct expr **e, vcc_type_t fmt) case CNUM: assert(fmt != VOID); if (fmt == BYTES) { - vcc_ByteVal(tl, &d); + vcc_ByteVal(tl, &vi); ERRCHK(tl); - e1 = vcc_mk_expr(BYTES, "%.1f", d); + e1 = vcc_mk_expr(BYTES, "%ju", (intmax_t)vi); } else { t = tl->t; vcc_NextToken(tl); diff --git a/lib/libvcc/vcc_utils.c b/lib/libvcc/vcc_utils.c index 7e4e00a99..3c6ac8c29 100644 --- a/lib/libvcc/vcc_utils.c +++ b/lib/libvcc/vcc_utils.c @@ -333,7 +333,7 @@ vcc_Duration(struct vcc *tl, double *d) /*--------------------------------------------------------------------*/ void -vcc_ByteVal(struct vcc *tl, double *d) +vcc_ByteVal(struct vcc *tl, VCL_INT *d) { double v, sc; @@ -355,7 +355,7 @@ vcc_ByteVal(struct vcc *tl, double *d) return; } vcc_NextToken(tl); - *d = v * sc; + *d = (VCL_INT)round((v * sc)); } /*--------------------------------------------------------------------*/ From phk at FreeBSD.org Wed May 19 12:29:07 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 12:29:07 +0000 (UTC) Subject: [master] 8cd84febf Enforce RFC8941 limits on INT/REAL et al. when converting to strings. Message-ID: <20210519122907.1FED7112620@lists.varnish-cache.org> commit 8cd84febfde4c168a9ef1b44b1baa7d573687ce0 Author: Poul-Henning Kamp Date: Wed May 19 12:28:05 2021 +0000 Enforce RFC8941 limits on INT/REAL et al. when converting to strings. diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c index 82cd54e53..4939e6dd5 100644 --- a/bin/varnishd/cache/cache_vrt.c +++ b/bin/varnishd/cache/cache_vrt.c @@ -829,6 +829,9 @@ VRT_INT_string(VRT_CTX, VCL_INT num) { CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + if (!VRT_INT_is_valid(num)) + VRT_fail(ctx, "INT overflow converting to string (%jX)", + (intmax_t)num); return (WS_Printf(ctx->ws, "%jd", (intmax_t)num)); } @@ -843,6 +846,8 @@ VRT_REAL_string(VRT_CTX, VCL_REAL num) { CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + if (!VRT_REAL_is_valid(num)) + VRT_fail(ctx, "REAL overflow converting to string (%e)", num); return (WS_Printf(ctx->ws, "%.3f", num)); } diff --git a/bin/varnishtest/tests/i00001.vtc b/bin/varnishtest/tests/i00001.vtc new file mode 100644 index 000000000..52c5a1a02 --- /dev/null +++ b/bin/varnishtest/tests/i00001.vtc @@ -0,0 +1,34 @@ +varnishtest "SF-decimal/SF-integer ranges" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -vcl+backend { + sub vcl_deliver { + if (req.http.foo) { + set resp.http.foo = obj.ttl * 10000000000; + } + if (req.http.bar) { + set resp.http.bar = storage.Transient.free_space * 1000000000000000; + } + } +} -start + +logexpect l1 -v v1 -g raw { + expect * 1001 VCL_Error "REAL overflow converting to string.*" + expect * 1004 VCL_Error "INT overflow converting to string.*" +} -start + +client c1 { + txreq -hdr "foo: 1" + rxresp +} -run + +client c1 { + txreq -hdr "bar: 1" + rxresp +} -run + +logexpect l1 -wait diff --git a/bin/varnishtest/tests/m00007.vtc b/bin/varnishtest/tests/m00007.vtc index 51840f5dd..784c887df 100644 --- a/bin/varnishtest/tests/m00007.vtc +++ b/bin/varnishtest/tests/m00007.vtc @@ -80,32 +80,32 @@ client c1 { expect resp.http.converted == 0 # VCL_INT_MAX - txreq -hdr "foo: 9007199254740991" \ - -hdr "bytes: 9007199254740991b" \ - -hdr "duration: 9007199254740991s" \ - -hdr "real: 9007199254740991" \ - -hdr "time: 9007199254740991" + txreq -hdr "foo: 999999999999999" \ + -hdr "bytes: 999999999999999b" \ + -hdr "duration: 999999999999.999s" \ + -hdr "real: 999999999999.999" \ + -hdr "time: 999999999999.999" rxresp expect resp.http.gtzero == true expect resp.http.ltzero == false expect resp.http.iszero == false - expect resp.http.converted == 9007199254740991 - expect resp.http.bytes == 9007199254740991 - expect resp.http.duration == 9007199254740991 - expect resp.http.real == 9007199254740991 - expect resp.http.time == 9007199254740991 + expect resp.http.converted == 999999999999999 + expect resp.http.bytes == 999999999999999 + expect resp.http.duration == 999999999999 + expect resp.http.real == 999999999999 + expect resp.http.time == 999999999999 # VCL_INT_MIN - txreq -hdr "foo: -9007199254740991" \ - -hdr "duration: -9007199254740991s" \ - -hdr "real: -9007199254740991" + txreq -hdr "foo: -999999999999999" \ + -hdr "duration: -999999999999s" \ + -hdr "real: -999999999999" rxresp expect resp.http.gtzero == false expect resp.http.ltzero == true expect resp.http.iszero == false - expect resp.http.converted == -9007199254740991 - expect resp.http.duration == -9007199254740991 - expect resp.http.real == -9007199254740991 + expect resp.http.converted == -999999999999999 + expect resp.http.duration == -999999999999 + expect resp.http.real == -999999999999 txreq -hdr "foo: bar" rxresp diff --git a/bin/varnishtest/tests/m00015.vtc b/bin/varnishtest/tests/m00015.vtc index 3e7974958..3aebc7e4d 100644 --- a/bin/varnishtest/tests/m00015.vtc +++ b/bin/varnishtest/tests/m00015.vtc @@ -95,38 +95,31 @@ client c1 { expect resp.http.converted == 0.000 # VCL_INT_MAX - txreq -hdr "foo: 9007199254740991" \ - -hdr "bytes: 9007199254740991b" \ - -hdr "duration: 9007199254740991s" \ - -hdr "integer: 9007199254740991" \ - -hdr "time: 9007199254740991" + txreq -hdr "foo: 999999999999.999" \ + -hdr "bytes: 999999999999b" \ + -hdr "duration: 999999999999.999s" \ + -hdr "integer: 999999999999.999" \ + -hdr "time: 999999999999.999" rxresp - expect resp.http.converted == 9007199254740991.000 - expect resp.http.bytes == 9007199254740991.000 - expect resp.http.duration == 9007199254740991.000 - expect resp.http.integer == 9007199254740991.000 - expect resp.http.time == 9007199254740991.000 + expect resp.http.converted == 999999999999.999 + expect resp.http.bytes == 999999999999.000 + expect resp.http.duration == 999999999999.999 + expect resp.http.integer == 999999999999.000 + expect resp.http.time == 999999999999.999 # VCL_INT_MIN - txreq -hdr "foo: -9007199254740991" \ - -hdr "duration: -9007199254740991s" \ - -hdr "integer: -9007199254740991" + txreq -hdr "foo: -999999999999.999" \ + -hdr "duration: -999999999999.999s" \ + -hdr "integer: -999999999999" rxresp - expect resp.http.converted == -9007199254740991.000 - expect resp.http.duration == -9007199254740991.000 - expect resp.http.integer == -9007199254740991.000 + expect resp.http.converted == -999999999999.999 + expect resp.http.duration == -999999999999.999 + expect resp.http.integer == -999999999999.000 txreq -hdr "foo: bar" rxresp expect resp.http.converted == 0.000 - txreq -hdr "foo: 9007199254740992" - rxresp - expect resp.http.converted == 9007199254740992.000 - - txreq -hdr "foo: -9007199254740992" - rxresp - expect resp.http.converted == -9007199254740992.000 } -run client c1 { txreq -hdr "nofb: NAN" diff --git a/bin/varnishtest/tests/v00020.vtc b/bin/varnishtest/tests/v00020.vtc index d5df60d4a..e29e4ff4f 100644 --- a/bin/varnishtest/tests/v00020.vtc +++ b/bin/varnishtest/tests/v00020.vtc @@ -430,8 +430,8 @@ varnish v1 -vcl+backend { set resp.http.bar = true == false; } sub vcl_deliver { - set resp.http.p = (0 + 9223372036854775807); - set resp.http.n = (0 - 9223372036854775807); + set resp.http.p = (0 + 999999999999999); + set resp.http.n = (0 - 999999999999999); if (resp.status == -(-200)) { set resp.http.o = -std.integer("-200"); } @@ -444,8 +444,8 @@ client c1 { expect resp.http.rst == "0" expect resp.http.foo == "true" expect resp.http.bar == "false" - expect resp.http.p == 9223372036854775807 - expect resp.http.n == -9223372036854775807 + expect resp.http.p == 999999999999999 + expect resp.http.n == -999999999999999 expect resp.http.o == 200 } -run diff --git a/vmod/vmod_std_conversions.c b/vmod/vmod_std_conversions.c index accc57f9d..838cd60af 100644 --- a/vmod/vmod_std_conversions.c +++ b/vmod/vmod_std_conversions.c @@ -319,14 +319,16 @@ vmod_time(VRT_CTX, struct VARGS(time)* a) VCL_INT v_matchproto_(td_std_real2integer) vmod_real2integer(VRT_CTX, VCL_REAL r, VCL_INT i) { + VCL_INT retval; + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - if (!isfinite(r)) + if (!VRT_REAL_is_valid(r)) return (i); - r = round(r); - if (r > VCL_INT_MAX || r < VCL_INT_MIN) + retval = round(r); + if (!VRT_INT_is_valid(r)) return (i); - return ((VCL_INT)r); + return (retval); } VCL_TIME v_matchproto_(td_std_real2time) From phk at FreeBSD.org Wed May 19 12:42:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 May 2021 12:42:06 +0000 (UTC) Subject: [master] db265fd03 Typo Message-ID: <20210519124207.0589511303D@lists.varnish-cache.org> commit db265fd03495cda0adc2a9c45724e4f5c7c312a8 Author: Poul-Henning Kamp Date: Wed May 19 12:41:17 2021 +0000 Typo diff --git a/vmod/vmod_std_conversions.c b/vmod/vmod_std_conversions.c index 838cd60af..1a4e9b3dc 100644 --- a/vmod/vmod_std_conversions.c +++ b/vmod/vmod_std_conversions.c @@ -325,8 +325,8 @@ vmod_real2integer(VRT_CTX, VCL_REAL r, VCL_INT i) if (!VRT_REAL_is_valid(r)) return (i); - retval = round(r); - if (!VRT_INT_is_valid(r)) + retval = (VCL_INT)round(r); + if (!VRT_INT_is_valid(retval)) return (i); return (retval); } From phk at FreeBSD.org Thu May 20 07:57:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 20 May 2021 07:57:05 +0000 (UTC) Subject: [master] 0abf0df57 RFC8941 parsing of VCL_BYTES. Message-ID: <20210520075705.B400F10D563@lists.varnish-cache.org> commit 0abf0df57f4ac5363685a7fe1e6275b3302aa0f0 Author: Poul-Henning Kamp Date: Thu May 20 07:55:45 2021 +0000 RFC8941 parsing of VCL_BYTES. diff --git a/bin/varnishtest/tests/r03308.vtc b/bin/varnishtest/tests/r03308.vtc index 144dcf8e9..115581be7 100644 --- a/bin/varnishtest/tests/r03308.vtc +++ b/bin/varnishtest/tests/r03308.vtc @@ -11,7 +11,10 @@ varnish v1 -vcl+backend { import std; sub vcl_deliver { - set resp.http.ts = std.real2time(std.real("10000000000000000000000", 0), now); + set resp.http.ts = std.real2time( + std.real("999999999999.999", 0) * + std.real("999999999999.999", 0), + now); } } -start diff --git a/include/vnum.h b/include/vnum.h index 93f797e72..09523f6f3 100644 --- a/include/vnum.h +++ b/include/vnum.h @@ -38,6 +38,13 @@ vtim_dur VNUM_duration(const char *p); double VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel); const char *VNUM_2bytes(const char *p, uintmax_t *r, uintmax_t rel); +#if 0 +int64_t SF_Parse_Integer(const char **ipp, const char **errtxt); +double SF_Parse_Decimal(const char **ipp, const char **errtxt); +#endif + +double SF_Parse_Number(const char **ipp, const char **errtxt); + #define VNUM_LEGAL_DURATION \ "Legal duration units are 'ms', 's', 'm', 'h', 'd', 'w' and 'y'" diff --git a/lib/libvarnish/vnum.c b/lib/libvarnish/vnum.c index 84b2acff8..207a16297 100644 --- a/lib/libvarnish/vnum.c +++ b/lib/libvarnish/vnum.c @@ -45,9 +45,106 @@ #include "vct.h" static const char err_miss_num[] = "Missing number"; +static const char err_fatnum[] = "Too may digits"; static const char err_invalid_num[] = "Invalid number"; static const char err_invalid_suff[] = "Invalid suffix"; +#define BAIL(txt) \ + do { \ + if (errtxt != NULL) \ + *errtxt = (txt); \ + errno = EINVAL; \ + return (0); \ + } while (0) + +static int64_t +sf_parse_int(const char **ipp, const char **errtxt, int maxdig) +{ + int64_t retval = 0; + int negative = 0, ndig = 0; + + AN(ipp); + AN(*ipp); + errno = 0; + while (vct_isows(*(*ipp))) + (*ipp)++; + if(*(*ipp) == '-') { + negative = 1; + (*ipp)++; + } + while (vct_isdigit(*(*ipp))) { + ndig++; + if (ndig > maxdig) + BAIL(err_fatnum); + retval *= 10; + retval += *(*ipp)++ - 0x30; + } + if (ndig == 0) + BAIL(negative ? err_invalid_num : err_miss_num); + if (negative) + retval = -retval; + return (retval); +} + +#if 0 + +int64_t +SF_Parse_Integer(const char **ipp, const char **errtxt) +{ + + return(sf_parse_int(ipp, errtxt, 15)); +} + +double +SF_Parse_Decimal(const char **ipp, const char **errtxt) +{ + double retval; + + retval = (double)sf_parse_int(ipp, errtxt, 12); + if (*(*ipp) != '.') + return (retval); + (*ipp)++; + if (!vct_isdigit(*(*ipp))) + return (retval); + retval += .1 * (*(*ipp)++ - 0x30); + if (!vct_isdigit(*(*ipp))) + return (retval); + retval += .01 * (*(*ipp)++ - 0x30); + if (!vct_isdigit(*(*ipp))) + return (retval); + retval += .001 * (*(*ipp)++ - 0x30); + if (vct_isdigit(*(*ipp))) + BAIL(err_fatnum); + return (retval); +} + +#endif + +double +SF_Parse_Number(const char **ipp, const char **errtxt) +{ + double retval; + + retval = (double)sf_parse_int(ipp, errtxt, 15); + if (*(*ipp) != '.') + return (retval); + if (retval < -999999999999 || retval > 999999999999) + BAIL(err_fatnum); + (*ipp)++; + if (!vct_isdigit(*(*ipp))) + return (retval); + retval += .1 * (*(*ipp)++ - 0x30); + if (!vct_isdigit(*(*ipp))) + return (retval); + retval += .01 * (*(*ipp)++ - 0x30); + if (!vct_isdigit(*(*ipp))) + return (retval); + retval += .001 * (*(*ipp)++ - 0x30); + if (vct_isdigit(*(*ipp))) + BAIL(err_fatnum); + return (retval); +} + /********************************************************************** * Convert (all of!) a string to a floating point number, and if we can * not, return NAN. @@ -175,7 +272,7 @@ VNUM_duration(const char *p) double VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel) { - double sc = 1.0; + double sc = 1.0, tmp; if (e == NULL) e = strchr(b, '\0'); @@ -196,6 +293,8 @@ VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel) case 't': case 'T': sc = exp2(40); b++; break; case 'p': case 'P': sc = exp2(50); b++; break; case 'b': case 'B': + if (modf(r, &tmp) != 0.0) + return (nan("")); break; default: return (nan("")); @@ -213,22 +312,26 @@ VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel) const char * VNUM_2bytes(const char *p, uintmax_t *r, uintmax_t rel) { - double fval; - const char *end; + double fval, tmp; + const char *errtxt; if (p == NULL || *p == '\0') return (err_miss_num); - fval = VNUMpfx(p, &end); - if (isnan(fval)) - return (err_invalid_num); + fval = SF_Parse_Number(&p, &errtxt); + if (errno) + return(errtxt); + if (fval < 0) + return(err_invalid_num); - if (end == NULL) { + if (*p == '\0') { + if (modf(fval, &tmp) != 0.0) + return (err_invalid_num); *r = (uintmax_t)fval; return (NULL); } - fval = VNUM_bytes_unit(fval, end, NULL, rel); + fval = VNUM_bytes_unit(fval, p, NULL, rel); if (isnan(fval)) return (err_invalid_suff); *r = (uintmax_t)round(fval); @@ -250,20 +353,21 @@ static struct test_case { { "1", (uintmax_t)0, (uintmax_t)1 }, { "1B", (uintmax_t)0, (uintmax_t)1<<0 }, { "1 B", (uintmax_t)0, (uintmax_t)1<<0 }, - { "1.3B", (uintmax_t)0, (uintmax_t)1 }, - { "1.7B", (uintmax_t)0, (uintmax_t)2 }, + { "1.3B", 0, 0, err_invalid_suff }, + { "1.7B", 0, 0, err_invalid_suff }, { "1024", (uintmax_t)0, (uintmax_t)1024 }, { "1k", (uintmax_t)0, (uintmax_t)1<<10 }, { "1kB", (uintmax_t)0, (uintmax_t)1<<10 }, + { "0.75kB", (uintmax_t)0, (uintmax_t)768 }, { "1.3kB", (uintmax_t)0, (uintmax_t)1331 }, - { "1.7kB", (uintmax_t)0, (uintmax_t)1741 }, + { "1.70kB", (uintmax_t)0, (uintmax_t)1741 }, { "1048576", (uintmax_t)0, (uintmax_t)1048576 }, { "1M", (uintmax_t)0, (uintmax_t)1<<20 }, { "1MB", (uintmax_t)0, (uintmax_t)1<<20 }, { "1.3MB", (uintmax_t)0, (uintmax_t)1363149 }, - { "1.7MB", (uintmax_t)0, (uintmax_t)1782579 }, + { "1.700MB", (uintmax_t)0, (uintmax_t)1782579 }, { "1073741824", (uintmax_t)0, (uintmax_t)1073741824 }, { "1G", (uintmax_t)0, (uintmax_t)1<<30 }, @@ -277,24 +381,25 @@ static struct test_case { { "1.3TB", (uintmax_t)0, (uintmax_t)1429365116109ULL }, { "1.7\tTB", (uintmax_t)0, (uintmax_t)1869169767219ULL }, - { "1125899906842624", (uintmax_t)0, (uintmax_t)1125899906842624ULL}, + { "999999999999999", (uintmax_t)0, (uintmax_t)999999999999999ULL}, + + { "1125899906842624", 0, 0, err_fatnum }, { "1P\t", (uintmax_t)0, (uintmax_t)1125899906842624ULL}, { "1PB ", (uintmax_t)0, (uintmax_t)1125899906842624ULL}, { "1.3 PB", (uintmax_t)0, (uintmax_t)1463669878895411ULL}, - // highest integers not rounded for double conversion - { "9007199254740988", (uintmax_t)0, (uintmax_t)9007199254740988ULL}, - { "9007199254740989", (uintmax_t)0, (uintmax_t)9007199254740989ULL}, - { "9007199254740990", (uintmax_t)0, (uintmax_t)9007199254740990ULL}, - { "9007199254740991", (uintmax_t)0, (uintmax_t)9007199254740991ULL}, - - { "1%", (uintmax_t)1024, (uintmax_t)10 }, + { "1.5%", (uintmax_t)1024, (uintmax_t)15 }, + { "1.501%", (uintmax_t)1024, (uintmax_t)15 }, { "2%", (uintmax_t)1024, (uintmax_t)20 }, { "3%", (uintmax_t)1024, (uintmax_t)31 }, /* Check the error checks */ { "", 0, 0, err_miss_num }, - { "m", 0, 0, err_invalid_num }, + { "-1", 0, 0, err_invalid_num }, + { "1.3", 0, 0, err_invalid_num }, + { "1.5011%", 0, 0, err_fatnum }, + { "-", 0, 0, err_invalid_num }, + { "m", 0, 0, err_miss_num }, { "4%", 0, 0, err_invalid_suff }, { "3*", 0, 0, err_invalid_suff }, @@ -340,6 +445,8 @@ main(int argc, char *argv[]) (void)argc; + setbuf(stdout, NULL); + setbuf(stderr, NULL); for (p = vec; *p != NULL; p++) { e = *p; d1 = VNUM(e + 1); From phk at FreeBSD.org Fri May 21 08:08:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 May 2021 08:08:06 +0000 (UTC) Subject: [master] 8db663a8b Split number handling into separate function. Message-ID: <20210521080806.C73A910E165@lists.varnish-cache.org> commit 8db663a8b8b7e3d33d0bd7760ea39102442bffe4 Author: Poul-Henning Kamp Date: Fri May 21 06:56:16 2021 +0000 Split number handling into separate function. diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 99f42df66..020ada098 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -672,6 +672,38 @@ vcc_Eval_SymFunc(struct vcc *tl, struct expr **e, struct token *t, } } +/*-------------------------------------------------------------------- + */ + +static void +vcc_number(struct vcc *tl, struct expr **e, vcc_type_t fmt, const char *sign) +{ + VCL_INT vi; + struct expr *e1; + struct token *t; + + assert(fmt != VOID); + if (fmt == BYTES) { + vcc_ByteVal(tl, &vi); + ERRCHK(tl); + e1 = vcc_mk_expr(BYTES, "%ju", (intmax_t)vi); + } else { + t = tl->t; + vcc_NextToken(tl); + if (tl->t->tok == ID) { + e1 = vcc_mk_expr(DURATION, "(%s%.*s) * %g", + sign, PF(t), vcc_DurationUnit(tl)); + ERRCHK(tl); + } else if (fmt == REAL || t->tok == FNUM) { + e1 = vcc_mk_expr(REAL, "%s%.*s", sign, PF(t)); + } else { + e1 = vcc_mk_expr(INT, "%s%.*s", sign, PF(t)); + } + } + e1->constant = EXPR_CONST; + *e = e1; +} + /*-------------------------------------------------------------------- * SYNTAX: * Expr5: @@ -689,7 +721,6 @@ vcc_expr5(struct vcc *tl, struct expr **e, vcc_type_t fmt) const char *ip, *sign; struct token *t; struct symbol *sym; - VCL_INT vi; sign = ""; *e = NULL; @@ -803,26 +834,7 @@ vcc_expr5(struct vcc *tl, struct expr **e, vcc_type_t fmt) /* FALLTHROUGH */ case FNUM: case CNUM: - assert(fmt != VOID); - if (fmt == BYTES) { - vcc_ByteVal(tl, &vi); - ERRCHK(tl); - e1 = vcc_mk_expr(BYTES, "%ju", (intmax_t)vi); - } else { - t = tl->t; - vcc_NextToken(tl); - if (tl->t->tok == ID) { - e1 = vcc_mk_expr(DURATION, "(%s%.*s) * %g", - sign, PF(t), vcc_DurationUnit(tl)); - ERRCHK(tl); - } else if (fmt == REAL || t->tok == FNUM) { - e1 = vcc_mk_expr(REAL, "%s%.*s", sign, PF(t)); - } else { - e1 = vcc_mk_expr(INT, "%s%.*s", sign, PF(t)); - } - } - e1->constant = EXPR_CONST; - *e = e1; + vcc_number(tl, e, fmt, sign); return; case CBLOB: e1 = vcc_new_expr(BLOB); From phk at FreeBSD.org Fri May 21 08:08:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 May 2021 08:08:06 +0000 (UTC) Subject: [master] 20a88a86d Use only RFC8941 compliant numbers. Message-ID: <20210521080806.DBBDE10E168@lists.varnish-cache.org> commit 20a88a86dc680a89ee159ea6b1bd30c0baeb0dfe Author: Poul-Henning Kamp Date: Fri May 21 07:58:17 2021 +0000 Use only RFC8941 compliant numbers. diff --git a/bin/varnishtest/tests/m00016.vtc b/bin/varnishtest/tests/m00016.vtc index 201682eb7..04b58f236 100644 --- a/bin/varnishtest/tests/m00016.vtc +++ b/bin/varnishtest/tests/m00016.vtc @@ -22,9 +22,7 @@ varnish v1 -vcl+backend { set resp.http.x-xyzzy = std.integer( std.real(req.http.foo, 2.0), 2); - # Representation of 9e99, which is larger than what fits in the - # 128bit integers on $exotic_platform. - set resp.http.x-int-fallback = std.real2integer(9000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.000, 2); + set resp.http.x-int-fallback = std.real2integer(123456789012.345 * 1000.0 * 10, 2); } } -start From phk at FreeBSD.org Fri May 21 08:27:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 May 2021 08:27:05 +0000 (UTC) Subject: [master] 9e561a693 Switch more of VCC to RFC8941 number parsing Message-ID: <20210521082705.6298610EB9C@lists.varnish-cache.org> commit 9e561a693460c774b7b5ba66d0cf972e1b468b55 Author: Poul-Henning Kamp Date: Fri May 21 08:26:27 2021 +0000 Switch more of VCC to RFC8941 number parsing diff --git a/include/vnum.h b/include/vnum.h index 09523f6f3..181ecee7c 100644 --- a/include/vnum.h +++ b/include/vnum.h @@ -38,11 +38,8 @@ vtim_dur VNUM_duration(const char *p); double VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel); const char *VNUM_2bytes(const char *p, uintmax_t *r, uintmax_t rel); -#if 0 int64_t SF_Parse_Integer(const char **ipp, const char **errtxt); double SF_Parse_Decimal(const char **ipp, const char **errtxt); -#endif - double SF_Parse_Number(const char **ipp, const char **errtxt); #define VNUM_LEGAL_DURATION \ diff --git a/lib/libvarnish/vnum.c b/lib/libvarnish/vnum.c index 207a16297..7397eea41 100644 --- a/lib/libvarnish/vnum.c +++ b/lib/libvarnish/vnum.c @@ -86,8 +86,6 @@ sf_parse_int(const char **ipp, const char **errtxt, int maxdig) return (retval); } -#if 0 - int64_t SF_Parse_Integer(const char **ipp, const char **errtxt) { @@ -118,8 +116,6 @@ SF_Parse_Decimal(const char **ipp, const char **errtxt) return (retval); } -#endif - double SF_Parse_Number(const char **ipp, const char **errtxt) { diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index 5c4139153..f97e5b46d 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -433,7 +433,7 @@ void Resolve_Sockaddr(struct vcc *tl, const char *host, const char *defport, double vcc_DurationUnit(struct vcc *); void vcc_ByteVal(struct vcc *, VCL_INT *); void vcc_Duration(struct vcc *tl, double *); -unsigned vcc_UintVal(struct vcc *tl); +uint64_t vcc_UintVal(struct vcc *tl); int vcc_IsFlag(struct vcc *tl); int vcc_IsFlagRaw(struct vcc *, const struct token *, const struct token *); char *vcc_Dup_be(const char *b, const char *e); diff --git a/lib/libvcc/vcc_utils.c b/lib/libvcc/vcc_utils.c index 3c6ac8c29..51ed651c7 100644 --- a/lib/libvcc/vcc_utils.c +++ b/lib/libvcc/vcc_utils.c @@ -281,39 +281,51 @@ vcc_DurationUnit(struct vcc *tl) * The tokenizer made sure we only get digits. */ -unsigned +uint64_t vcc_UintVal(struct vcc *tl) { - unsigned d = 0; - const char *p; + const char *p, *errtxt; + int64_t retval; if (tl->t->tok != CNUM) { Expect(tl, CNUM); - } else { - for (p = tl->t->b; p < tl->t->e; p++) { - d *= 10; - d += *p - '0'; - } - vcc_NextToken(tl); + return (0); + } + p = tl->t->b; + retval = SF_Parse_Integer(&p, &errtxt); + if (errno) { + VSB_printf(tl->sb, "Bad UINT: %s\n", errtxt); + vcc_ErrWhere(tl, tl->t); + return (0); + } + if (retval < 0) { + VSB_printf(tl->sb, "UINT cannot be negative\n"); + vcc_ErrWhere(tl, tl->t); + return (0); } - return (d); + vcc_NextToken(tl); + return (retval); } static double vcc_DoubleVal(struct vcc *tl) { - const size_t l = tl->t->e - tl->t->b; - char buf[l + 1]; + const char *p, *errtxt; + double retval; if (tl->t->tok != CNUM && tl->t->tok != FNUM) { Expect(tl, CNUM); return (0); } - assert(tl->t->tok == CNUM || tl->t->tok == FNUM); - memcpy(buf, tl->t->b, l); + p = tl->t->b; + retval = SF_Parse_Decimal(&p, &errtxt); + if (errno) { + VSB_printf(tl->sb, "Bad REAL: %s\n", errtxt); + vcc_ErrWhere(tl, tl->t); + return (0); + } vcc_NextToken(tl); - buf[l] = '\0'; - return (strtod(buf, NULL)); + return (retval); } /*--------------------------------------------------------------------*/ From phk at FreeBSD.org Fri May 21 10:42:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 May 2021 10:42:05 +0000 (UTC) Subject: [master] b908baebc RFC8941 renovation of BYTES Message-ID: <20210521104205.2D014112309@lists.varnish-cache.org> commit b908baebca31d422522cbd592e4ea5f7dc1cdbe4 Author: Poul-Henning Kamp Date: Fri May 21 10:41:41 2021 +0000 RFC8941 renovation of BYTES diff --git a/bin/varnishtest/tests/v00033.vtc b/bin/varnishtest/tests/v00033.vtc index 5d9b1e6b4..c5884fd9b 100644 --- a/bin/varnishtest/tests/v00033.vtc +++ b/bin/varnishtest/tests/v00033.vtc @@ -43,7 +43,7 @@ varnish v1 -syntax 4.0 -errvcl {Expected BYTES unit (B, KB, MB...) got '"X"'} { } } } -varnish v1 -syntax 4.0 -errvcl {Unknown BYTES unit 'X'} { +varnish v1 -syntax 4.0 -errvcl {Unknown BYTES unit} { sub vcl_recv { if (storage.Transient.free_space > 4 X) { } diff --git a/include/vnum.h b/include/vnum.h index 181ecee7c..859d8cbb0 100644 --- a/include/vnum.h +++ b/include/vnum.h @@ -35,7 +35,8 @@ double VNUM(const char *p); double VNUMpfx(const char *p, const char **e); vtim_dur VNUM_duration_unit(vtim_dur r, const char *b, const char *e); vtim_dur VNUM_duration(const char *p); -double VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel); +int64_t VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel, + const char **errtxt); const char *VNUM_2bytes(const char *p, uintmax_t *r, uintmax_t rel); int64_t SF_Parse_Integer(const char **ipp, const char **errtxt); @@ -44,6 +45,3 @@ double SF_Parse_Number(const char **ipp, const char **errtxt); #define VNUM_LEGAL_DURATION \ "Legal duration units are 'ms', 's', 'm', 'h', 'd', 'w' and 'y'" - -#define VNUM_LEGAL_BYTES \ - "Legal byte units are 'B', 'KB', 'MB', 'GB', 'TB' and 'PB'" diff --git a/lib/libvarnish/vnum.c b/lib/libvarnish/vnum.c index 7397eea41..7f44c6228 100644 --- a/lib/libvarnish/vnum.c +++ b/lib/libvarnish/vnum.c @@ -47,7 +47,9 @@ static const char err_miss_num[] = "Missing number"; static const char err_fatnum[] = "Too may digits"; static const char err_invalid_num[] = "Invalid number"; -static const char err_invalid_suff[] = "Invalid suffix"; +static const char err_unknown_bytes[] = + "Unknown BYTES unit of measurement ([KMGTP][B])"; +static const char err_fractional_bytes[] = "Fractional BYTES not allowed"; #define BAIL(txt) \ do { \ @@ -265,18 +267,27 @@ VNUM_duration(const char *p) /**********************************************************************/ -double -VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel) +int64_t +VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel, + const char **errtxt) { double sc = 1.0, tmp; + AN(b); + AN(errtxt); + errno = 0; if (e == NULL) e = strchr(b, '\0'); while (b < e && vct_issp(*b)) b++; - if (b == e) - return (nan("")); + if (b == e) { + if (modf(r, &tmp) != 0.0) { + *errtxt = err_fractional_bytes; + errno = EINVAL; + } + return ((int64_t)trunc(sc * r)); + } if (rel != 0 && *b == '%') { r *= rel * 0.01; @@ -289,26 +300,34 @@ VNUM_bytes_unit(double r, const char *b, const char *e, uintmax_t rel) case 't': case 'T': sc = exp2(40); b++; break; case 'p': case 'P': sc = exp2(50); b++; break; case 'b': case 'B': - if (modf(r, &tmp) != 0.0) - return (nan("")); + if (modf(r, &tmp) != 0.0) { + *errtxt = err_fractional_bytes; + errno = EINVAL; + return (0); + } break; default: - return (nan("")); + *errtxt = err_unknown_bytes; + errno = EINVAL; + return (0); } if (b < e && (*b == 'b' || *b == 'B')) b++; } while (b < e && vct_issp(*b)) b++; - if (b < e) - return (nan("")); - return (sc * r); + if (b < e) { + *errtxt = err_unknown_bytes; + errno = EINVAL; + return (0); + } + return ((int64_t)trunc(sc * r)); } const char * VNUM_2bytes(const char *p, uintmax_t *r, uintmax_t rel) { - double fval, tmp; + double fval; const char *errtxt; if (p == NULL || *p == '\0') @@ -320,16 +339,9 @@ VNUM_2bytes(const char *p, uintmax_t *r, uintmax_t rel) if (fval < 0) return(err_invalid_num); - if (*p == '\0') { - if (modf(fval, &tmp) != 0.0) - return (err_invalid_num); - *r = (uintmax_t)fval; - return (NULL); - } - - fval = VNUM_bytes_unit(fval, p, NULL, rel); - if (isnan(fval)) - return (err_invalid_suff); + fval = VNUM_bytes_unit(fval, p, NULL, rel, &errtxt); + if (errno) + return (errtxt); *r = (uintmax_t)round(fval); return (NULL); } @@ -349,32 +361,32 @@ static struct test_case { { "1", (uintmax_t)0, (uintmax_t)1 }, { "1B", (uintmax_t)0, (uintmax_t)1<<0 }, { "1 B", (uintmax_t)0, (uintmax_t)1<<0 }, - { "1.3B", 0, 0, err_invalid_suff }, - { "1.7B", 0, 0, err_invalid_suff }, + { "1.3B", 0, 0, err_fractional_bytes }, + { "1.7B", 0, 0, err_fractional_bytes }, { "1024", (uintmax_t)0, (uintmax_t)1024 }, { "1k", (uintmax_t)0, (uintmax_t)1<<10 }, { "1kB", (uintmax_t)0, (uintmax_t)1<<10 }, { "0.75kB", (uintmax_t)0, (uintmax_t)768 }, { "1.3kB", (uintmax_t)0, (uintmax_t)1331 }, - { "1.70kB", (uintmax_t)0, (uintmax_t)1741 }, + { "1.70kB", (uintmax_t)0, (uintmax_t)1740 }, { "1048576", (uintmax_t)0, (uintmax_t)1048576 }, { "1M", (uintmax_t)0, (uintmax_t)1<<20 }, { "1MB", (uintmax_t)0, (uintmax_t)1<<20 }, - { "1.3MB", (uintmax_t)0, (uintmax_t)1363149 }, + { "1.3MB", (uintmax_t)0, (uintmax_t)1363148 }, { "1.700MB", (uintmax_t)0, (uintmax_t)1782579 }, { "1073741824", (uintmax_t)0, (uintmax_t)1073741824 }, { "1G", (uintmax_t)0, (uintmax_t)1<<30 }, { "1GB", (uintmax_t)0, (uintmax_t)1<<30 }, { "1.3GB", (uintmax_t)0, (uintmax_t)1395864371 }, - { "1.7GB", (uintmax_t)0, (uintmax_t)1825361101 }, + { "1.7GB", (uintmax_t)0, (uintmax_t)1825361100 }, { "1099511627776", (uintmax_t)0, (uintmax_t)1099511627776ULL }, { "1T", (uintmax_t)0, (uintmax_t)1<<40 }, { "1TB", (uintmax_t)0, (uintmax_t)1<<40 }, - { "1.3TB", (uintmax_t)0, (uintmax_t)1429365116109ULL }, + { "1.3TB", (uintmax_t)0, (uintmax_t)1429365116108ULL }, { "1.7\tTB", (uintmax_t)0, (uintmax_t)1869169767219ULL }, { "999999999999999", (uintmax_t)0, (uintmax_t)999999999999999ULL}, @@ -387,17 +399,17 @@ static struct test_case { { "1.5%", (uintmax_t)1024, (uintmax_t)15 }, { "1.501%", (uintmax_t)1024, (uintmax_t)15 }, { "2%", (uintmax_t)1024, (uintmax_t)20 }, - { "3%", (uintmax_t)1024, (uintmax_t)31 }, + { "3%", (uintmax_t)1024, (uintmax_t)30 }, /* Check the error checks */ { "", 0, 0, err_miss_num }, { "-1", 0, 0, err_invalid_num }, - { "1.3", 0, 0, err_invalid_num }, + { "1.3", 0, 0, err_fractional_bytes}, { "1.5011%", 0, 0, err_fatnum }, { "-", 0, 0, err_invalid_num }, { "m", 0, 0, err_miss_num }, - { "4%", 0, 0, err_invalid_suff }, - { "3*", 0, 0, err_invalid_suff }, + { "4%", 0, 0, err_unknown_bytes }, + { "3*", 0, 0, err_unknown_bytes }, /* TODO: add more */ diff --git a/lib/libvcc/vcc_utils.c b/lib/libvcc/vcc_utils.c index 51ed651c7..47f6ab00c 100644 --- a/lib/libvcc/vcc_utils.c +++ b/lib/libvcc/vcc_utils.c @@ -347,10 +347,22 @@ vcc_Duration(struct vcc *tl, double *d) void vcc_ByteVal(struct vcc *tl, VCL_INT *d) { - double v, sc; + double v; + VCL_INT retval; + const char *p, *errtxt; - v = vcc_DoubleVal(tl); - ERRCHK(tl); + if (tl->t->tok != CNUM && tl->t->tok != FNUM) { + Expect(tl, CNUM); + return; + } + p = tl->t->b; + v = SF_Parse_Number(&p, &errtxt); + if (errno) { + VSB_printf(tl->sb, "Bad BYTES: %s\n", errtxt); + vcc_ErrWhere(tl, tl->t); + return; + } + vcc_NextToken(tl); if (tl->t->tok != ID) { VSB_cat(tl->sb, "Expected BYTES unit (B, KB, MB...) got "); vcc_ErrToken(tl, tl->t); @@ -358,16 +370,15 @@ vcc_ByteVal(struct vcc *tl, VCL_INT *d) vcc_ErrWhere(tl, tl->t); return; } - sc = VNUM_bytes_unit(1.0, tl->t->b, tl->t->e, 0); - if (isnan(sc)) { - VSB_cat(tl->sb, "Unknown BYTES unit "); + retval = VNUM_bytes_unit(v, tl->t->b, tl->t->e, 0, &errtxt); + if (errno) { + VSB_cat(tl->sb, errtxt); vcc_ErrToken(tl, tl->t); - VSB_printf(tl->sb, "\n%s\n", VNUM_LEGAL_BYTES); vcc_ErrWhere(tl, tl->t); return; } vcc_NextToken(tl); - *d = (VCL_INT)round((v * sc)); + *d = retval; } /*--------------------------------------------------------------------*/ diff --git a/vmod/vmod_std_conversions.c b/vmod/vmod_std_conversions.c index 1a4e9b3dc..ab7f61cb4 100644 --- a/vmod/vmod_std_conversions.c +++ b/vmod/vmod_std_conversions.c @@ -113,6 +113,7 @@ vmod_bytes(VRT_CTX, struct VARGS(bytes) *a) uintmax_t r; VCL_REAL rr; int nargs; + const char *errtxt; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); @@ -121,10 +122,11 @@ vmod_bytes(VRT_CTX, struct VARGS(bytes) *a) if (!onearg(ctx, "bytes", nargs)) return (0); - if (a->valid_s && - VNUM_2bytes(a->s, &r, 0) == NULL && - r <= VCL_BYTES_MAX) - return ((VCL_BYTES)r); + if (a->valid_s) { + errtxt = VNUM_2bytes(a->s, &r, 0); + if (errtxt == NULL && r <= VCL_BYTES_MAX) + return ((VCL_BYTES)r); + } if (a->valid_real && !isnan(a->real) && a->real >= 0) { rr = trunc(a->real); From phk at FreeBSD.org Mon May 24 15:07:07 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 24 May 2021 15:07:07 +0000 (UTC) Subject: [master] 6c533a053 About our IBM S390x CI client Message-ID: <20210524150707.2F7C411BBBB@lists.varnish-cache.org> commit 6c533a053c16d6bf76dc9ac0da07a30f487b90f6 Author: Poul-Henning Kamp Date: Mon May 24 14:43:29 2021 +0000 About our IBM S390x CI client diff --git a/doc/sphinx/phk/index.rst b/doc/sphinx/phk/index.rst index cbe4ff314..8897a0662 100644 --- a/doc/sphinx/phk/index.rst +++ b/doc/sphinx/phk/index.rst @@ -13,6 +13,7 @@ You may or may not want to know what Poul-Henning thinks. .. toctree:: :maxdepth: 1 + legacy.rst ip_address.rst vdd19q3.rst quic.rst diff --git a/doc/sphinx/phk/legacy.rst b/doc/sphinx/phk/legacy.rst new file mode 100644 index 000000000..914861c05 --- /dev/null +++ b/doc/sphinx/phk/legacy.rst @@ -0,0 +1,81 @@ +.. + Copyright (c) 2021 Varnish Software AS + SPDX-License-Identifier: BSD-2-Clause + See LICENSE file for full text of license + +.. _phk_legacy: + +========= +Legacy-.* +========= + +In the middle of an otherwise pleaseant conversation recently, the +other person suddenly burst out that "Varnish was part of our +legacy software." +That stung a bit. + +But fair enough: They *have* been running varnish since 2009 or so. + +Neither Raymond's "New Hacker's Dictionary", nor the legacy publication +it tried to replace, Kelly-Bootle's "The Devils DP Dictionary", define +"legacy software". The former because the collective "we" did not +bother with utter trivialities such as invoicing, the latter because +back then people didnt abandon software. + +Tomorrow I will be sitting in a small hut in the middle of a field, +trying to figure out what somebody could possibly have been thinking +about, when 10 years ago they claimed to implement "V.42" while +also using their own HDLC frame layout. + +"V.42" and "HDLC" are also a legacy at this point, but chances are +you have used it: That was the hot way to do error-correction on +modems, when dialing into a BBS or ISP in the 1990ies. + +I guess I should say "legacy-modems" ? + +Big-endianess, storing they bytes the sensible way for hex-dumps, is +rapidly becoming legacy, as the final old HP and SUN irons are +finally become eWaste. + +Objectively there is no difference of merit between little-endian +and big-endian, the most successful computers of all time fall +equally into one or the other, and the consolidation towards +little-endian is more "It's not important" than anything +else. + +But we still have a bit of code which cares about endianess +in Varnish, in particular in the imported `zlib` code. + +For a while I ran a CI client on a WLAN access point with a +big-endian MIPS-processor. But with only 128MB RAM the spurious +error rate made too much noise. + +Nothing has been proclaimed "Legacy" more often and with more force +than the IBM mainframe, but they are still around, keeping the books +balanced, like they have for half a century. + +And because they were born with variable length data types, IBM +mainframes are big-endian, and because we in Varnish care about +portability, you can also run Varnish Cache on your IBM mainframe. + +Thanks mainly to Ingvar, there are "s390x" architecture Varnish +packages if you need them. + +So I reached out to IBM's FOSS-outreach people and asked if we could +borrow a cup of mainframe to run a CI-client, and before I knew it, +the Varnish Cache Project had access to a virtual s390x machine somewhere +in IBM's cloud. + +For once "In the Cloud" *literally* means "On somebody's mainframe" :-) + +I'm not up to date on IBM Mainframe technology, the last one I used +was a 3090 in 1989, so I have no idea how much performance +IBM has allocated to us, on what kind of hardware, or what it might +cost. + +But it runs a full CI iteration on Varnish Cache in 3 minutes flat, +making it one of the fastest CI-clients we have. + +Thanks IBM! + +Poul-Henning, 2021-05-24 From phk at FreeBSD.org Mon May 24 20:10:08 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 24 May 2021 20:10:08 +0000 (UTC) Subject: [master] 8b2893131 Typos & polish Message-ID: <20210524201008.6B27264934@lists.varnish-cache.org> commit 8b2893131ecc81b783fbe8ec0ba549527e068d45 Author: Poul-Henning Kamp Date: Mon May 24 20:08:41 2021 +0000 Typos & polish diff --git a/doc/sphinx/phk/legacy.rst b/doc/sphinx/phk/legacy.rst index 914861c05..d605a4fda 100644 --- a/doc/sphinx/phk/legacy.rst +++ b/doc/sphinx/phk/legacy.rst @@ -10,11 +10,12 @@ Legacy-.* ========= In the middle of an otherwise pleaseant conversation recently, the -other person suddenly burst out that "Varnish was part of our -legacy software." +other person suddenly burst out that *"Varnish was part of our +legacy software."* + That stung a bit. -But fair enough: They *have* been running varnish since 2009 or so. +But fair enough: They have been running varnish since 2009 or so. Neither Raymond's "New Hacker's Dictionary", nor the legacy publication it tried to replace, Kelly-Bootle's "The Devils DP Dictionary", define @@ -33,32 +34,31 @@ modems, when dialing into a BBS or ISP in the 1990ies. I guess I should say "legacy-modems" ? -Big-endianess, storing they bytes the sensible way for hex-dumps, is +Big-endianess, storing the bytes the sensible way for hex-dumps, is rapidly becoming legacy, as the final old HP and SUN irons are finally become eWaste. Objectively there is no difference of merit between little-endian -and big-endian, the most successful computers of all time fall -equally into one or the other, and the consolidation towards -little-endian is more "It's not important" than anything -else. +and big-endian, the most successful computers architectures of all +time picked equally one or the other, and the consolidation towards +little-endian is driven more by *"It's actually not that important"* +than by anything else. But we still have a bit of code which cares about endianess in Varnish, in particular in the imported `zlib` code. For a while I ran a CI client on a WLAN access point with a big-endian MIPS-processor. But with only 128MB RAM the spurious -error rate made too much noise. +error rate caused too much noise. -Nothing has been proclaimed "Legacy" more often and with more force +Nothing has been proclaimed "Legacy" more often and with more force, than the IBM mainframe, but they are still around, keeping the books -balanced, like they have for half a century. +balanced, as they have for half a century. And because they were born with variable length data types, IBM mainframes are big-endian, and because we in Varnish care about -portability, you can also run Varnish Cache on your IBM mainframe. - -Thanks mainly to Ingvar, there are "s390x" architecture Varnish +portability, you can also run Varnish Cache on your IBM mainframe: +Thanks to Ingvar, there are "s390x" architecture Varnish packages if you need them. So I reached out to IBM's FOSS-outreach people and asked if we could From guillaume at varnish-software.com Mon May 24 21:54:08 2021 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 24 May 2021 21:54:08 +0000 (UTC) Subject: [master] 36cc65c09 [cci] lower make paralellism Message-ID: <20210524215408.EE69C10099A@lists.varnish-cache.org> commit 36cc65c09fb87aaa0ed3d27614cdd0482d8aa954 Author: Guillaume Quintard Date: Mon May 24 14:52:01 2021 -0700 [cci] lower make paralellism under pressure, tests get slow and time out diff --git a/.circleci/config.yml b/.circleci/config.yml index 61e9a4726..53457fee0 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -260,7 +260,7 @@ jobs: << parameters.extra_conf >> sudo -u varnish \ --preserve-env=ASAN_OPTIONS,LSAN_OPTIONS,TSAN_OPTIONS,UBSAN_OPTIONS \ - make distcheck VERBOSE=1 -j 12 -k \ + make distcheck VERBOSE=1 -j 4 -k \ DISTCHECK_CONFIGURE_FLAGS="<< pipeline.parameters.configure_args >> \ << parameters.extra_conf >>" From dridi.boukelmoune at gmail.com Wed May 26 09:47:07 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 26 May 2021 09:47:07 +0000 (UTC) Subject: [master] 2608ea8bd vtc: Fix h5 on my machine Message-ID: <20210526094707.D62B7115935@lists.varnish-cache.org> commit 2608ea8bd4b7d7ccc4eb0b4848e38491cdc8d948 Author: Dridi Boukelmoune Date: Wed May 26 11:41:53 2021 +0200 vtc: Fix h5 on my machine Ever since my system upgraded haproxy to 2.3.10 this test has consistently timed out. While that would be a breaking change involving the independent vtest project too, I think the VTC syslog spec would work better with something like: expect skip facility.level regex Where skip could be uint, * or ? similar to how logexpect works, and both facility and level could also be * to be non-specific. For now, let's hope this does not break the test suite for anyone else. diff --git a/bin/varnishtest/tests/h00005.vtc b/bin/varnishtest/tests/h00005.vtc index b6281040e..52f1d8315 100644 --- a/bin/varnishtest/tests/h00005.vtc +++ b/bin/varnishtest/tests/h00005.vtc @@ -9,8 +9,6 @@ server s1 { } -start syslog S1 -level notice -bind "${listen_addr}" { - recv - recv recv info expect ~ \"dip\":\"${h1_fe_1_addr}\" } -start From nils.goroll at uplex.de Wed May 26 12:33:05 2021 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 May 2021 12:33:05 +0000 (UTC) Subject: [master] 4dbdd8a89 Update vnum.c Message-ID: <20210526123305.5F7FF119F14@lists.varnish-cache.org> commit 4dbdd8a897304d85ba9cc9fd33e46eefe65ed18e Author: Johan Hendriks Date: Wed May 26 14:23:42 2021 +0200 Update vnum.c Small typo diff --git a/lib/libvarnish/vnum.c b/lib/libvarnish/vnum.c index 7f44c6228..2cf665480 100644 --- a/lib/libvarnish/vnum.c +++ b/lib/libvarnish/vnum.c @@ -45,7 +45,7 @@ #include "vct.h" static const char err_miss_num[] = "Missing number"; -static const char err_fatnum[] = "Too may digits"; +static const char err_fatnum[] = "Too many digits"; static const char err_invalid_num[] = "Invalid number"; static const char err_unknown_bytes[] = "Unknown BYTES unit of measurement ([KMGTP][B])"; From phk at FreeBSD.org Fri May 28 08:32:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 28 May 2021 08:32:05 +0000 (UTC) Subject: [master] 810dcd936 Enforce the RFC8941 number ranges centrally. Message-ID: <20210528083205.F03E4114F7A@lists.varnish-cache.org> commit 810dcd9368c2218e4716480d641dfb9857aa7c4e Author: Poul-Henning Kamp Date: Fri May 28 08:31:35 2021 +0000 Enforce the RFC8941 number ranges centrally. diff --git a/bin/varnishtest/tests/i00001.vtc b/bin/varnishtest/tests/i00001.vtc index 52c5a1a02..a56300eb7 100644 --- a/bin/varnishtest/tests/i00001.vtc +++ b/bin/varnishtest/tests/i00001.vtc @@ -1,17 +1,38 @@ varnishtest "SF-decimal/SF-integer ranges" +varnish v1 -errvcl {Too many digits for integer.} { + sub vcl_recv { set req.http.foo = 1234567890123456; } +} + +varnish v1 -errvcl {Too many digits for real.} { + sub vcl_recv { set req.http.foo = 1234567890123.; } +} + +varnish v1 -errvcl {Too many digits for real.} { + sub vcl_recv { set req.http.foo = 123456789012.1234; } +} + +varnish v1 -errvcl {Too many digits for real.} { + sub vcl_recv { set req.http.foo = 0.1234; } +} + server s1 { rxreq txresp } -start varnish v1 -vcl+backend { + sub vcl_recv { + set req.http.foo1 = 123456789012345; + set req.http.foo2 = 123456789012.; + set req.http.foo3 = 123456789012.123; + } sub vcl_deliver { if (req.http.foo) { set resp.http.foo = obj.ttl * 10000000000; } if (req.http.bar) { - set resp.http.bar = storage.Transient.free_space * 1000000000000000; + set resp.http.bar = storage.Transient.free_space * 10000000 * 100000000; } } } -start diff --git a/lib/libvcc/vcc_token.c b/lib/libvcc/vcc_token.c index cc8392196..05f8d3627 100644 --- a/lib/libvcc/vcc_token.c +++ b/lib/libvcc/vcc_token.c @@ -437,6 +437,41 @@ vcc_delim_token(struct vcc *tl, struct source *sp, const char *p, return (1); } +/*-------------------------------------------------------------------- + * Lex a number, either CNUM or FNUM. + * We enforce the RFC8941 restrictions on number of digits here. + */ + +static const char * +vcc_lex_number(struct vcc *tl, struct source *sp, const char *p) +{ + const char *q, *r; + + for (q = p; q < sp->e; q++) + if (!vct_isdigit(*q)) + break; + if (*q != '.') { + vcc_addtoken(tl, CNUM, sp, p, q); + if (q - p > 15) { + VSB_cat(tl->sb, "Too many digits for integer.\n"); + vcc_ErrWhere(tl, tl->t); + return (NULL); + } + return (q); + } + r = ++q; + for (; r < sp->e; r++) + if (!vct_isdigit(*r)) + break; + vcc_addtoken(tl, FNUM, sp, p, r); + if (q - p > 13 || r - q > 3) { + VSB_cat(tl->sb, "Too many digits for real.\n"); + vcc_ErrWhere(tl, tl->t); + return(NULL); + } + return (r); +} + /*-------------------------------------------------------------------- * Lexical analysis and token generation */ @@ -588,19 +623,9 @@ vcc_Lexer(struct vcc *tl, struct source *sp) /* Match numbers { [0-9]+ } */ if (vct_isdigit(*p)) { - for (q = p; q < sp->e; q++) - if (!vct_isdigit(*q)) - break; - if (*q != '.') { - vcc_addtoken(tl, CNUM, sp, p, q); - p = q; - continue; - } - for (++q; q < sp->e; q++) - if (!vct_isdigit(*q)) - break; - vcc_addtoken(tl, FNUM, sp, p, q); - p = q; + p = vcc_lex_number(tl, sp, p); + if (p == NULL) + return; continue; } vcc_addtoken(tl, EOI, sp, p, p + 1); From dridi.boukelmoune at gmail.com Fri May 28 08:59:04 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 08:59:04 +0000 (UTC) Subject: [master] e1e52264a varnishtest: New tunnel command Message-ID: <20210528085904.E4CF5115BB8@lists.varnish-cache.org> commit e1e52264a3638bb3cea3a1551966e0702eefb351 Author: Dridi Boukelmoune Date: Fri Jul 31 18:14:26 2020 +0200 varnishtest: New tunnel command The tunnel acts like a client and a server and forwards bytes between two parties. It can then pause in the middle of a session and control how much data can go in either direction. Combined with barriers it can be used to trigger socket timeouts, possibly in the middle of protocol frames. A tunnel works with two threads: one to execute its spec and one to poll both parties and transfer data accordingly. The two threads synchronize via basic pthread primitives and follow a very simplistic state machine: ACCEPT -> RUNNING <-> PAUSED -> SPEC DONE -> POLL DONE -> STOPPED Despite being simplistic, the coordination of two threads would likely complicate the implementation of a `-repeat` action, but it is still possible to `start` a tunnel again as shown in the c106 test case that exercises basic coverage involving a varnish instance. Usage is documented in the vtc(7) manual. diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am index 30cea03ad..fae47f4a5 100644 --- a/bin/varnishtest/Makefile.am +++ b/bin/varnishtest/Makefile.am @@ -48,6 +48,7 @@ varnishtest_SOURCES = \ vtc_sess.c \ vtc_subr.c \ vtc_syslog.c \ + vtc_tunnel.c \ vtc_varnish.c varnishtest_LDADD = \ diff --git a/bin/varnishtest/cmds.h b/bin/varnishtest/cmds.h index 17a64408c..dccdcf5bb 100644 --- a/bin/varnishtest/cmds.h +++ b/bin/varnishtest/cmds.h @@ -53,6 +53,7 @@ CMD_TOP(process) CMD_TOP(server) CMD_TOP(setenv) CMD_TOP(syslog) +CMD_TOP(tunnel) #ifdef VTEST_WITH_VTC_VARNISH CMD_TOP(varnish) #endif diff --git a/bin/varnishtest/tests/a00021.vtc b/bin/varnishtest/tests/a00021.vtc new file mode 100644 index 000000000..c7c175e79 --- /dev/null +++ b/bin/varnishtest/tests/a00021.vtc @@ -0,0 +1,32 @@ +varnishtest "tunnel basic coverage" + +barrier b1 cond 2 +barrier b2 cond 2 +barrier b3 cond 2 + +server s1 { + rxreq + barrier b2 sync + barrier b3 sync + txresp +} -start + +tunnel t1 -connect "${s1_sock}" { + pause + barrier b1 sync + send 10 + resume + barrier b2 sync + pause + barrier b3 sync + recv 10 + resume +} -start + +client c1 -connect "${t1_sock}" { + barrier b1 sync + txreq + rxresp +} -run + +tunnel t1 -wait diff --git a/bin/varnishtest/tests/c00106.vtc b/bin/varnishtest/tests/c00106.vtc new file mode 100644 index 000000000..0461e04f4 --- /dev/null +++ b/bin/varnishtest/tests/c00106.vtc @@ -0,0 +1,49 @@ +varnishtest "tunnel basic coverage" + +barrier b1 cond 2 -cyclic +barrier b2 sock 2 -cyclic +barrier b3 sock 2 -cyclic + +server s1 { + rxreq + txresp +} -start + +varnish v1 -vcl+backend { + import vtc; + sub vcl_recv { + vtc.barrier_sync("${b2_sock}"); + vtc.barrier_sync("${b3_sock}"); + } +} -start + +tunnel t1 { + pause + barrier b1 sync + send 10 + delay 0.1 + send 15 + resume + barrier b2 sync + pause + barrier b3 sync + recv 10 + delay 0.1 + recv 15 + # automatic resumption here +} -start + +client c1 -connect "${t1_sock}" { + barrier b1 sync + txreq + rxresp + expect resp.status == 200 +} -run + +tunnel t1 -wait + +# Same scenario, but wait for c1 _after_ t1 +tunnel t1 -start +client c1 -start +tunnel t1 -wait +client c1 -wait diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c index f6d696471..7319d4d43 100644 --- a/bin/varnishtest/vtc.c +++ b/bin/varnishtest/vtc.c @@ -533,6 +533,7 @@ exec_file(const char *fn, const char *script, const char *tmpdir, init_macro(); init_server(); init_syslog(); + init_tunnel(); vsb = VSB_new_auto(); AN(vsb); diff --git a/bin/varnishtest/vtc.h b/bin/varnishtest/vtc.h index 046df86d0..d2b449359 100644 --- a/bin/varnishtest/vtc.h +++ b/bin/varnishtest/vtc.h @@ -85,6 +85,7 @@ extern const char *default_listen_addr; void init_server(void); void init_syslog(void); +void init_tunnel(void); /* Sessions */ struct vtc_sess *Sess_New(struct vtclog *vl, const char *name); @@ -147,6 +148,8 @@ void start_h2(struct http *hp); void stop_h2(struct http *hp); void b64_settings(const struct http *hp, const char *s); +cmd_f cmd_tunnel; + /* vtc_gzip.c */ void vtc_gzip(struct http *, const char *, char **, long *); void vtc_gunzip(struct http *, char *, long *); diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c new file mode 100644 index 000000000..41ce3c775 --- /dev/null +++ b/bin/varnishtest/vtc_tunnel.c @@ -0,0 +1,758 @@ +/*- + * Copyright (c) 2020 Varnish Software + * All rights reserved. + * + * Author: Dridi Boukelmoune + * + * SPDX-License-Identifier: BSD-2-Clause + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include "config.h" + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "vtc.h" + +#include "vtcp.h" + +/* SECTION: tunnel tunnel + * + * The goal of a tunnel is to help control the data transfer between two + * parties, for example to trigger socket timeouts in the middle of protocol + * frames, without the need to change how both parties are implemented. + * + * A tunnel accepts a connection and then connects on behalf of the source to + * the desired destination. Once both connections are established the tunnel + * will transfer bytes unchanged between the source and destination. Transfer + * can be interrupted, usually with the help of synchronization methods like + * barriers. Once the transfer is paused, it is possible to let a specific + * amount of bytes move in either direction. + * + * SECTION: tunnel.args Arguments + * + * \-start + * Start the tunnel in background, processing the last given + * specification. + * + * \-wait + * Block until the thread finishes. + * + * \-listen STRING + * Dictate the listening socket for the server. STRING is of the form + * "IP PORT", or "HOST PORT". + * + * Listens by defaults to a local random port. + * + * \-connect STRING + * Indicate the server to connect to. STRING is also of the form + * "IP PORT", or "HOST PORT". + * + * Connects by default to a varnish instance called ``v1``. + * + * SECTION: tunnel.spec Specification + * + * The specification contains a list of tunnel commands that can be combined + * with barriers and delays. For example:: + * + * tunnel t1 { + * barrier b1 sync + * pause + * delay 1 + * send 42 + * barrier b2 sync + * resume + * } -start + * + * If one end of the tunnel is closed before the end of the specification + * the test case will fail. A specification that ends in a paused state will + * implicitely resume the tunnel. + */ + +enum tunnel_state_e { + TUNNEL_ACCEPT, + TUNNEL_RUNNING, + TUNNEL_PAUSED, + TUNNEL_SPEC_DONE, + TUNNEL_POLL_DONE, + TUNNEL_STOPPED, +}; + +struct tunnel_lane { + char buf[1024]; + ssize_t buf_len; + size_t wrk_len; + int *rfd; + int *wfd; +}; + +struct tunnel { + unsigned magic; +#define TUNNEL_MAGIC 0x55286619 + char *name; + struct vtclog *vl; + VTAILQ_ENTRY(tunnel) list; + enum tunnel_state_e state; + + char *spec; + + char connect[256]; + int csock; + const char *caddr; + + char listen[256]; + int lsock; + char laddr[32]; + char lport[32]; + + int asock; + + struct tunnel_lane send_lane[1]; + struct tunnel_lane recv_lane[1]; + + pthread_mutex_t mtx; /* state and lanes->*_len */ + pthread_cond_t cond; + pthread_t tspec; + pthread_t tpoll; +}; + +static pthread_mutex_t tunnel_mtx; + +static VTAILQ_HEAD(, tunnel) tunnels = VTAILQ_HEAD_INITIALIZER(tunnels); + +/********************************************************************** + * Is the tunnel still operating? + */ + +static unsigned +tunnel_is_open(struct tunnel *t) +{ + unsigned is_open; + + AZ(pthread_mutex_lock(&t->mtx)); + is_open = (t->send_lane->buf_len >= 0 && t->recv_lane->buf_len >= 0); + AZ(pthread_mutex_unlock(&t->mtx)); + return (is_open); +} + +/********************************************************************** + * SECTION: tunnel.spec.pause + * + * pause + * Wait for in-flight bytes to be transferred and pause the tunnel. + * + * The tunnel must be running. + */ + +static void +cmd_tunnel_pause(CMD_ARGS) +{ + struct tunnel *t; + + CAST_OBJ_NOTNULL(t, priv, TUNNEL_MAGIC); + AZ(av[1]); + + if (!tunnel_is_open(t)) + vtc_fatal(vl, "Tunnel already closed"); + + AZ(pthread_mutex_lock(&t->mtx)); + if (t->state == TUNNEL_PAUSED) { + AZ(pthread_mutex_unlock(&t->mtx)); + vtc_fatal(vl, "Tunnel already paused"); + WRONG("unreachable"); + } + assert(t->state == TUNNEL_RUNNING); + t->state = TUNNEL_PAUSED; + AZ(pthread_cond_signal(&t->cond)); + AZ(pthread_cond_wait(&t->cond, &t->mtx)); + AZ(pthread_mutex_unlock(&t->mtx)); +} + +/********************************************************************** + * SECTION: tunnel.spec.send + * + * send NUMBER + * Wait until NUMBER bytes are transferred from source to + * destination. + * + * The tunnel must be paused, it remains paused afterwards. + */ + +static void +cmd_tunnel_send(CMD_ARGS) +{ + struct tunnel *t; + unsigned len; + + CAST_OBJ_NOTNULL(t, priv, TUNNEL_MAGIC); + AN(av[1]); + AZ(av[2]); + + len = atoi(av[1]); + + if (!tunnel_is_open(t)) + vtc_fatal(vl, "Tunnel already closed"); + + AZ(pthread_mutex_lock(&t->mtx)); + if (t->state == TUNNEL_RUNNING) { + AZ(pthread_mutex_unlock(&t->mtx)); + vtc_fatal(vl, "Tunnel still running"); + WRONG("unreachable"); + } + assert(t->state == TUNNEL_PAUSED); + AZ(t->send_lane->wrk_len); + AZ(t->recv_lane->wrk_len); + if (!strcmp(av[0], "send")) + t->send_lane->wrk_len = len; + else + t->recv_lane->wrk_len = len; + AZ(pthread_cond_signal(&t->cond)); + AZ(pthread_cond_wait(&t->cond, &t->mtx)); + AZ(pthread_mutex_unlock(&t->mtx)); +} + +/********************************************************************** + * SECTION: tunnel.spec.recv + * + * recv NUMBER + * Wait until NUMBER bytes are transferred from destination to + * source. + * + * The tunnel must be paused, it remains paused afterwards. + */ + +static void +cmd_tunnel_recv(CMD_ARGS) +{ + + cmd_tunnel_send(av, priv, vl); +} + +/********************************************************************** + * SECTION: tunnel.spec.resume + * + * resume + * Resume the transfer of bytes in both directions. + * + * The tunnel must be paused. + */ + +static void +cmd_tunnel_resume(CMD_ARGS) +{ + struct tunnel *t; + + CAST_OBJ_NOTNULL(t, priv, TUNNEL_MAGIC); + AZ(av[1]); + + if (!tunnel_is_open(t)) + vtc_fatal(vl, "Tunnel already closed"); + + AZ(pthread_mutex_lock(&t->mtx)); + if (t->state == TUNNEL_RUNNING) { + AZ(pthread_mutex_unlock(&t->mtx)); + vtc_fatal(vl, "Tunnel already running"); + WRONG("unreachable"); + } + assert(t->state == TUNNEL_PAUSED); + t->state = TUNNEL_RUNNING; + AZ(pthread_cond_signal(&t->cond)); + AZ(pthread_mutex_unlock(&t->mtx)); +} + +const struct cmds tunnel_cmds[] = { +#define CMD_TUNNEL(n) { #n, cmd_tunnel_##n }, + CMD_TUNNEL(pause) + CMD_TUNNEL(send) + CMD_TUNNEL(recv) + CMD_TUNNEL(resume) +#undef CMD_TUNNEL + { NULL, NULL } +}; + +/********************************************************************** + * Tunnel poll thread + */ + +static void +tunnel_read(struct tunnel *t, struct vtclog *vl, struct pollfd *pfd, + struct tunnel_lane *lane) +{ + size_t len; + ssize_t res; + enum tunnel_state_e state; + + assert(pfd->fd == *lane->rfd); + if (!(pfd->revents & POLLIN)) + return; + + AZ(pthread_mutex_lock(&t->mtx)); + AZ(lane->buf_len); + len = lane->wrk_len; + state = t->state; + AZ(pthread_mutex_unlock(&t->mtx)); + + if (len == 0 && state == TUNNEL_PAUSED) + return; + + if (len == 0 || len > sizeof lane->buf) + len = sizeof lane->buf; + + res = read(pfd->fd, lane->buf, len); + if (res < 0) + vtc_fatal(vl, "Read failed: %s", strerror(errno)); + + AZ(pthread_mutex_lock(&t->mtx)); + lane->buf_len = (res == 0) ? -1 : res; + AZ(pthread_mutex_unlock(&t->mtx)); +} + +static void +tunnel_write(struct tunnel *t, struct vtclog *vl, struct tunnel_lane *lane, + const char *action) +{ + const char *p; + ssize_t res, l; + + p = lane->buf; + AZ(pthread_mutex_lock(&t->mtx)); + l = lane->buf_len; + AZ(pthread_mutex_unlock(&t->mtx)); + + if (l > 0) + vtc_log(vl, 3, "%s %zd bytes", action, l); + while (l > 0) { + res = write(*lane->wfd, p, l); + if (res <= 0) + vtc_fatal(vl, "Write failed: %s", strerror(errno)); + l -= res; + p += res; + } + + AZ(pthread_mutex_lock(&t->mtx)); + if (lane->wrk_len > 0 && lane->buf_len != -1) { + assert(lane->buf_len >= 0); + assert(lane->wrk_len >= lane->buf_len); + lane->wrk_len -= lane->buf_len; + } + lane->buf_len = l; + AZ(pthread_mutex_unlock(&t->mtx)); +} + +static void * +tunnel_poll_thread(void *priv) +{ + struct tunnel *t; + struct vtclog *vl; + struct pollfd pfd[2]; + enum tunnel_state_e state; + int res; + + CAST_OBJ_NOTNULL(t, priv, TUNNEL_MAGIC); + + vl = vtc_logopen("%s", t->name); + pthread_cleanup_push(vtc_logclose, vl); + + while (tunnel_is_open(t) && !vtc_stop) { + AZ(pthread_mutex_lock(&t->mtx)); + /* NB: can be woken up by `tunnel tX -wait` */ + while (t->state == TUNNEL_ACCEPT) + AZ(pthread_cond_wait(&t->cond, &t->mtx)); + state = t->state; + AZ(pthread_mutex_unlock(&t->mtx)); + + assert(state < TUNNEL_POLL_DONE); + + memset(pfd, 0, sizeof pfd); + pfd[0].fd = *t->send_lane->rfd; + pfd[1].fd = *t->recv_lane->rfd; + pfd[0].events = POLLIN; + pfd[1].events = POLLIN; + res = poll(pfd, 2, 100); + if (res == -1) + vtc_fatal(vl, "Poll failed: %s", strerror(errno)); + + tunnel_read(t, vl, &pfd[0], t->send_lane); + tunnel_read(t, vl, &pfd[1], t->recv_lane); + + AZ(pthread_mutex_lock(&t->mtx)); + if (t->state == TUNNEL_PAUSED && t->send_lane->wrk_len == 0 && + t->recv_lane->wrk_len == 0) { + AZ(t->send_lane->buf_len); + AZ(t->recv_lane->buf_len); + AZ(pthread_cond_signal(&t->cond)); + AZ(pthread_cond_wait(&t->cond, &t->mtx)); + } + AZ(pthread_mutex_unlock(&t->mtx)); + + tunnel_write(t, vl, t->send_lane, "Sending"); + tunnel_write(t, vl, t->recv_lane, "Receiving"); + } + + AZ(pthread_mutex_lock(&t->mtx)); + if (t->state != TUNNEL_SPEC_DONE && !vtc_stop) { + AZ(pthread_cond_signal(&t->cond)); + AZ(pthread_cond_wait(&t->cond, &t->mtx)); + } + AZ(pthread_mutex_unlock(&t->mtx)); + + pthread_cleanup_pop(0); + vtc_logclose(vl); + t->state = TUNNEL_POLL_DONE; + return (NULL); +} + +/********************************************************************** + * Tunnel spec thread + */ + +static void +tunnel_accept(struct tunnel *t, struct vtclog *vl) +{ + struct vsb *vsb; + const char *addr, *err; + int afd, cfd; + + CHECK_OBJ_NOTNULL(t, TUNNEL_MAGIC); + assert(t->lsock >= 0); + assert(t->asock < 0); + assert(t->csock < 0); + assert(t->state == TUNNEL_ACCEPT); + + vtc_log(vl, 4, "Accepting"); + afd = accept(t->lsock, NULL, NULL); + if (afd < 0) + vtc_fatal(vl, "Accept failed: %s", strerror(errno)); + vtc_log(vl, 3, "Accepted socket fd is %d", afd); + + vsb = macro_expand(vl, t->connect); + AN(vsb); + addr = VSB_data(vsb); + + cfd = VTCP_open(addr, NULL, 10., &err); + if (cfd < 0) + vtc_fatal(vl, "Failed to open %s: %s", addr, err); + vtc_log(vl, 3, "Connected socket fd is %d", cfd); + VSB_destroy(&vsb); + + VTCP_blocking(afd); + VTCP_blocking(cfd); + + AZ(pthread_mutex_lock(&t->mtx)); + t->asock = afd; + t->csock = cfd; + t->send_lane->buf_len = 0; + t->send_lane->wrk_len = 0; + t->recv_lane->buf_len = 0; + t->recv_lane->wrk_len = 0; + t->state = TUNNEL_RUNNING; + AZ(pthread_cond_signal(&t->cond)); + AZ(pthread_mutex_unlock(&t->mtx)); +} + +static void * +tunnel_spec_thread(void *priv) +{ + struct tunnel *t; + struct vtclog *vl; + enum tunnel_state_e state; + + CAST_OBJ_NOTNULL(t, priv, TUNNEL_MAGIC); + AN(*t->connect); + + vl = vtc_logopen("%s", t->name); + vtc_log_set_cmd(vl, tunnel_cmds); + pthread_cleanup_push(vtc_logclose, vl); + + tunnel_accept(t, vl); + parse_string(vl, t, t->spec); + + AZ(pthread_mutex_lock(&t->mtx)); + state = t->state; + AZ(pthread_mutex_unlock(&t->mtx)); + + if (state == TUNNEL_PAUSED && !vtc_stop) + parse_string(vl, t, "resume"); + + AZ(pthread_mutex_lock(&t->mtx)); + t->state = TUNNEL_SPEC_DONE; + AZ(pthread_cond_signal(&t->cond)); + AZ(pthread_mutex_unlock(&t->mtx)); + + vtc_log(vl, 2, "Ending"); + pthread_cleanup_pop(0); + vtc_logclose(vl); + return (NULL); +} + +/********************************************************************** + * Tunnel management + */ + +static struct tunnel * +tunnel_new(const char *name) +{ + struct tunnel *t; + + ALLOC_OBJ(t, TUNNEL_MAGIC); + AN(t); + REPLACE(t->name, name); + t->vl = vtc_logopen("%s", name); + AN(t->vl); + + t->state = TUNNEL_STOPPED; + bprintf(t->connect, "%s", "${v1_sock}"); + bprintf(t->listen, "%s", "127.0.0.1 0"); + t->csock = -1; + t->lsock = -1; + t->asock = -1; + t->send_lane->rfd = &t->asock; + t->send_lane->wfd = &t->csock; + t->recv_lane->rfd = &t->csock; + t->recv_lane->wfd = &t->asock; + AZ(pthread_mutex_init(&t->mtx, NULL)); + AZ(pthread_cond_init(&t->cond, NULL)); + AZ(pthread_mutex_lock(&tunnel_mtx)); + VTAILQ_INSERT_TAIL(&tunnels, t, list); + AZ(pthread_mutex_unlock(&tunnel_mtx)); + return (t); +} + +static void +tunnel_delete(struct tunnel *t) +{ + + CHECK_OBJ_NOTNULL(t, TUNNEL_MAGIC); + assert(t->asock < 0); + assert(t->csock < 0); + if (t->lsock >= 0) + VTCP_close(&t->lsock); + macro_undef(t->vl, t->name, "addr"); + macro_undef(t->vl, t->name, "port"); + macro_undef(t->vl, t->name, "sock"); + vtc_logclose(t->vl); + (void)pthread_mutex_destroy(&t->mtx); + (void)pthread_cond_destroy(&t->cond); + free(t->name); + FREE_OBJ(t); +} + +/********************************************************************** + * Tunnel listen + */ + +static void +tunnel_listen(struct tunnel *t) +{ + const char *err; + + if (t->lsock >= 0) + VTCP_close(&t->lsock); + t->lsock = VTCP_listen_on(t->listen, "0", 1, &err); + if (err != NULL) + vtc_fatal(t->vl, + "Tunnel listen address (%s) cannot be resolved: %s", + t->listen, err); + assert(t->lsock > 0); + VTCP_myname(t->lsock, t->laddr, sizeof t->laddr, + t->lport, sizeof t->lport); + macro_def(t->vl, t->name, "addr", "%s", t->laddr); + macro_def(t->vl, t->name, "port", "%s", t->lport); + macro_def(t->vl, t->name, "sock", "%s %s", t->laddr, t->lport); + /* Record the actual port, and reuse it on subsequent starts */ + bprintf(t->listen, "%s %s", t->laddr, t->lport); +} + +/********************************************************************** + * Start the tunnel thread + */ + +static void +tunnel_start(struct tunnel *t) +{ + + CHECK_OBJ_NOTNULL(t, TUNNEL_MAGIC); + vtc_log(t->vl, 2, "Starting tunnel"); + tunnel_listen(t); + vtc_log(t->vl, 1, "Listen on %s", t->listen); + assert(t->state == TUNNEL_STOPPED); + t->state = TUNNEL_ACCEPT; + t->send_lane->buf_len = 0; + t->send_lane->wrk_len = 0; + t->recv_lane->buf_len = 0; + t->recv_lane->wrk_len = 0; + AZ(pthread_create(&t->tpoll, NULL, tunnel_poll_thread, t)); + AZ(pthread_create(&t->tspec, NULL, tunnel_spec_thread, t)); +} + +/********************************************************************** + * Wait for tunnel thread to stop + */ + +static void +tunnel_wait(struct tunnel *t) +{ + void *res; + + CHECK_OBJ_NOTNULL(t, TUNNEL_MAGIC); + vtc_log(t->vl, 2, "Waiting for tunnel"); + + AZ(pthread_cond_signal(&t->cond)); + + AZ(pthread_join(t->tspec, &res)); + if (res == PTHREAD_CANCELED && !vtc_stop) + vtc_fatal(t->vl, "Tunnel spec canceled"); + if (res != NULL && !vtc_stop) + vtc_fatal(t->vl, "Tunnel spec returned \"%p\"", res); + + AZ(pthread_join(t->tpoll, &res)); + if (res == PTHREAD_CANCELED && !vtc_stop) + vtc_fatal(t->vl, "Tunnel poll canceled"); + if (res != NULL && !vtc_stop) + vtc_fatal(t->vl, "Tunnel poll returned \"%p\"", res); + + if (t->csock >= 0) + VTCP_close(&t->csock); + if (t->asock >= 0) + VTCP_close(&t->asock); + t->tspec = 0; + t->tpoll = 0; + t->state = TUNNEL_STOPPED; +} + +/********************************************************************** + * Reap tunnel + */ + +static void +tunnel_reset(void) +{ + struct tunnel *t; + enum tunnel_state_e state; + + while (1) { + AZ(pthread_mutex_lock(&tunnel_mtx)); + t = VTAILQ_FIRST(&tunnels); + CHECK_OBJ_ORNULL(t, TUNNEL_MAGIC); + if (t != NULL) + VTAILQ_REMOVE(&tunnels, t, list); + AZ(pthread_mutex_unlock(&tunnel_mtx)); + if (t == NULL) + break; + + AZ(pthread_mutex_lock(&t->mtx)); + state = t->state; + if (state < TUNNEL_POLL_DONE) + (void)pthread_cancel(t->tpoll); + if (state < TUNNEL_SPEC_DONE) + (void)pthread_cancel(t->tspec); + AZ(pthread_mutex_unlock(&t->mtx)); + + if (state != TUNNEL_STOPPED) + tunnel_wait(t); + tunnel_delete(t); + } +} + +/********************************************************************** + * Tunnel command dispatch + */ + +void +cmd_tunnel(CMD_ARGS) +{ + struct tunnel *t; + + (void)priv; + + if (av == NULL) { + /* Reset and free */ + tunnel_reset(); + return; + } + + AZ(strcmp(av[0], "tunnel")); + av++; + + VTC_CHECK_NAME(vl, av[0], "Tunnel", 't'); + + AZ(pthread_mutex_lock(&tunnel_mtx)); + VTAILQ_FOREACH(t, &tunnels, list) + if (!strcmp(t->name, av[0])) + break; + AZ(pthread_mutex_unlock(&tunnel_mtx)); + if (t == NULL) + t = tunnel_new(av[0]); + CHECK_OBJ_NOTNULL(t, TUNNEL_MAGIC); + av++; + + for (; *av != NULL; av++) { + if (vtc_error) + break; + if (!strcmp(*av, "-wait")) { + if (t->state == TUNNEL_STOPPED) + vtc_fatal(t->vl, "Tunnel not -started"); + tunnel_wait(t); + continue; + } + + /* Don't mess with a running tunnel */ + if (t->state != TUNNEL_STOPPED) + tunnel_wait(t); + + assert(t->state == TUNNEL_STOPPED); + if (!strcmp(*av, "-connect")) { + bprintf(t->connect, "%s", av[1]); + av++; + continue; + } + if (!strcmp(*av, "-listen")) { + bprintf(t->listen, "%s", av[1]); + av++; + continue; + } + if (!strcmp(*av, "-start")) { + tunnel_start(t); + continue; + } + if (**av == '-') + vtc_fatal(t->vl, "Unknown tunnel argument: %s", *av); + t->spec = *av; + } +} + +void +init_tunnel(void) +{ + + AZ(pthread_mutex_init(&tunnel_mtx, NULL)); +} diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am index a85815071..8327b478f 100644 --- a/doc/sphinx/Makefile.am +++ b/doc/sphinx/Makefile.am @@ -157,16 +157,17 @@ include/vsl-tags.rst: $(top_builddir)/lib/libvarnishapi/vsl2rst mv -f ${@}_ ${@} BUILT_SOURCES += include/vsl-tags.rst -VTCSYN_SRC = $(top_srcdir)/bin/varnishtest/vtc.c \ - $(top_srcdir)/bin/varnishtest/vtc_barrier.c \ - $(top_srcdir)/bin/varnishtest/vtc_haproxy.c \ - $(top_srcdir)/bin/varnishtest/vtc_http.c \ - $(top_srcdir)/bin/varnishtest/vtc_http2.c \ - $(top_srcdir)/bin/varnishtest/vtc_logexp.c \ - $(top_srcdir)/bin/varnishtest/vtc_misc.c \ - $(top_srcdir)/bin/varnishtest/vtc_process.c \ - $(top_srcdir)/bin/varnishtest/vtc_syslog.c \ - $(top_srcdir)/bin/varnishtest/vtc_varnish.c +VTCSYN_SRC = \ + $(top_srcdir)/bin/varnishtest/vtc_barrier.c \ + $(top_srcdir)/bin/varnishtest/vtc_haproxy.c \ + $(top_srcdir)/bin/varnishtest/vtc_http.c \ + $(top_srcdir)/bin/varnishtest/vtc_http2.c \ + $(top_srcdir)/bin/varnishtest/vtc_logexp.c \ + $(top_srcdir)/bin/varnishtest/vtc_misc.c \ + $(top_srcdir)/bin/varnishtest/vtc_process.c \ + $(top_srcdir)/bin/varnishtest/vtc_syslog.c \ + $(top_srcdir)/bin/varnishtest/vtc_tunnel.c \ + $(top_srcdir)/bin/varnishtest/vtc_varnish.c include/vtc-syntax.rst: vtc-syntax.py $(VTCSYN_SRC) $(AM_V_GEN) $(PYTHON) $(top_srcdir)/doc/sphinx/vtc-syntax.py $(VTCSYN_SRC) > ${@}_ @mv -f ${@}_ ${@} From dridi.boukelmoune at gmail.com Fri May 28 08:59:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 08:59:05 +0000 (UTC) Subject: [master] 1e2e40ce6 vtc: Allow a tunnel to start already paused Message-ID: <20210528085905.076F2115BBB@lists.varnish-cache.org> commit 1e2e40ce6488910411fc674aa9cb40d615e6bf97 Author: Dridi Boukelmoune Date: Fri May 28 09:46:02 2021 +0200 vtc: Allow a tunnel to start already paused To cut down synchronization needs for the very beginning of a session. diff --git a/bin/varnishtest/tests/a00021.vtc b/bin/varnishtest/tests/a00021.vtc index c7c175e79..9eb158582 100644 --- a/bin/varnishtest/tests/a00021.vtc +++ b/bin/varnishtest/tests/a00021.vtc @@ -30,3 +30,17 @@ client c1 -connect "${t1_sock}" { } -run tunnel t1 -wait + +server s2 { + rxreq + txresp +} -start + +tunnel t2 -connect "${s2_sock}" { + resume +} -start+pause + +client c2 -connect "${t2_sock}" { + txreq + rxresp +} -run diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index 41ce3c775..4f6f24c7b 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -64,6 +64,9 @@ * Start the tunnel in background, processing the last given * specification. * + * \-start+pause + * Start the tunnel, but already paused. + * * \-wait * Block until the thread finishes. * @@ -122,6 +125,7 @@ struct tunnel { struct vtclog *vl; VTAILQ_ENTRY(tunnel) list; enum tunnel_state_e state; + unsigned start_paused; char *spec; @@ -474,7 +478,11 @@ tunnel_accept(struct tunnel *t, struct vtclog *vl) t->send_lane->wrk_len = 0; t->recv_lane->buf_len = 0; t->recv_lane->wrk_len = 0; - t->state = TUNNEL_RUNNING; + if (t->start_paused) { + t->state = TUNNEL_PAUSED; + t->start_paused = 0; + } else + t->state = TUNNEL_RUNNING; AZ(pthread_cond_signal(&t->cond)); AZ(pthread_mutex_unlock(&t->mtx)); } @@ -614,6 +622,15 @@ tunnel_start(struct tunnel *t) AZ(pthread_create(&t->tspec, NULL, tunnel_spec_thread, t)); } +static void +tunnel_start_pause(struct tunnel *t) +{ + + CHECK_OBJ_NOTNULL(t, TUNNEL_MAGIC); + t->start_paused = 1; + tunnel_start(t); +} + /********************************************************************** * Wait for tunnel thread to stop */ @@ -744,6 +761,10 @@ cmd_tunnel(CMD_ARGS) tunnel_start(t); continue; } + if (!strcmp(*av, "-start+pause")) { + tunnel_start_pause(t); + continue; + } if (**av == '-') vtc_fatal(t->vl, "Unknown tunnel argument: %s", *av); t->spec = *av; From dridi.boukelmoune at gmail.com Fri May 28 09:28:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 09:28:05 +0000 (UTC) Subject: [master] 57cbd8450 varnishtest: Remove redundant WRONGs Message-ID: <20210528092805.58756116ADF@lists.varnish-cache.org> commit 57cbd8450682336196ccc3c5433e0b4829474d47 Author: Dridi Boukelmoune Date: Fri May 28 11:26:07 2021 +0200 varnishtest: Remove redundant WRONGs A wrong WRONG doesn't make it right for SunCC. diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index 4f6f24c7b..231fcc919 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -192,7 +192,6 @@ cmd_tunnel_pause(CMD_ARGS) if (t->state == TUNNEL_PAUSED) { AZ(pthread_mutex_unlock(&t->mtx)); vtc_fatal(vl, "Tunnel already paused"); - WRONG("unreachable"); } assert(t->state == TUNNEL_RUNNING); t->state = TUNNEL_PAUSED; @@ -230,7 +229,6 @@ cmd_tunnel_send(CMD_ARGS) if (t->state == TUNNEL_RUNNING) { AZ(pthread_mutex_unlock(&t->mtx)); vtc_fatal(vl, "Tunnel still running"); - WRONG("unreachable"); } assert(t->state == TUNNEL_PAUSED); AZ(t->send_lane->wrk_len); @@ -285,7 +283,6 @@ cmd_tunnel_resume(CMD_ARGS) if (t->state == TUNNEL_RUNNING) { AZ(pthread_mutex_unlock(&t->mtx)); vtc_fatal(vl, "Tunnel already running"); - WRONG("unreachable"); } assert(t->state == TUNNEL_PAUSED); t->state = TUNNEL_RUNNING; From dridi.boukelmoune at gmail.com Fri May 28 09:32:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 09:32:05 +0000 (UTC) Subject: [master] eda05508f varnishtest: Use the default listen address Message-ID: <20210528093205.B2128116DBC@lists.varnish-cache.org> commit eda05508feeb362b4b998e160f6e670e91d262f7 Author: Dridi Boukelmoune Date: Fri May 28 11:31:21 2021 +0200 varnishtest: Use the default listen address diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index 231fcc919..fc316fc35 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -536,7 +536,7 @@ tunnel_new(const char *name) t->state = TUNNEL_STOPPED; bprintf(t->connect, "%s", "${v1_sock}"); - bprintf(t->listen, "%s", "127.0.0.1 0"); + bprintf(t->listen, "%s", default_listen_addr); t->csock = -1; t->lsock = -1; t->asock = -1; From dridi.boukelmoune at gmail.com Fri May 28 10:17:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 10:17:05 +0000 (UTC) Subject: [master] 4334c501c varnishtest: Align tunnel's listen setup to server's Message-ID: <20210528101705.DBBF111810C@lists.varnish-cache.org> commit 4334c501cadea312c305d420b010a44725e6ae24 Author: Dridi Boukelmoune Date: Fri May 28 12:14:55 2021 +0200 varnishtest: Align tunnel's listen setup to server's diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index fc316fc35..418ce9195 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -43,6 +43,7 @@ #include "vtc.h" +#include "vsa.h" #include "vtcp.h" /* SECTION: tunnel tunnel @@ -578,6 +579,8 @@ tunnel_delete(struct tunnel *t) static void tunnel_listen(struct tunnel *t) { + char buf[vsa_suckaddr_len]; + struct suckaddr *sua; const char *err; if (t->lsock >= 0) @@ -588,13 +591,20 @@ tunnel_listen(struct tunnel *t) "Tunnel listen address (%s) cannot be resolved: %s", t->listen, err); assert(t->lsock > 0); - VTCP_myname(t->lsock, t->laddr, sizeof t->laddr, + sua = VSA_getsockname(t->lsock, buf, sizeof buf); + AN(sua); + VTCP_name(sua, t->laddr, sizeof t->laddr, t->lport, sizeof t->lport); + + /* Record the actual port, and reuse it on subsequent starts */ + if (VSA_Get_Proto(sua) == AF_INET) + bprintf(t->listen, "%s:%s", t->laddr, t->lport); + else + bprintf(t->listen, "[%s]:%s", t->laddr, t->lport); + macro_def(t->vl, t->name, "addr", "%s", t->laddr); macro_def(t->vl, t->name, "port", "%s", t->lport); macro_def(t->vl, t->name, "sock", "%s %s", t->laddr, t->lport); - /* Record the actual port, and reuse it on subsequent starts */ - bprintf(t->listen, "%s %s", t->laddr, t->lport); } /********************************************************************** From dridi.boukelmoune at gmail.com Fri May 28 10:17:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 10:17:05 +0000 (UTC) Subject: [master] f74ab7ab1 varnishtest: Whitespace OCD Message-ID: <20210528101705.F07E011810F@lists.varnish-cache.org> commit f74ab7ab109bea2b1277c394b27334ce5b35b8fe Author: Dridi Boukelmoune Date: Fri May 28 12:15:40 2021 +0200 varnishtest: Whitespace OCD diff --git a/bin/varnishtest/vtc_server.c b/bin/varnishtest/vtc_server.c index 87637ed73..77dcae8dc 100644 --- a/bin/varnishtest/vtc_server.c +++ b/bin/varnishtest/vtc_server.c @@ -165,7 +165,7 @@ server_listen_uds(struct server *s, const char **errp) errno = 0; if (unlink(s->listen) != 0 && errno != ENOENT) vtc_fatal(s->vl, "Could not unlink %s before bind: %s", - s->listen, strerror(errno)); + s->listen, strerror(errno)); /* * Temporarily set the umask to 0 to avoid issues with * permissions. From phk at FreeBSD.org Fri May 28 10:18:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 28 May 2021 10:18:05 +0000 (UTC) Subject: [master] 0f44918db Exercise the vcc_ErrWhere2() multiline code. Message-ID: <20210528101805.0716B118410@lists.varnish-cache.org> commit 0f44918db7b4a0f60e155f360703947f7758ba72 Author: Poul-Henning Kamp Date: Fri May 28 10:09:40 2021 +0000 Exercise the vcc_ErrWhere2() multiline code. diff --git a/bin/varnishtest/tests/m00008.vtc b/bin/varnishtest/tests/m00008.vtc index 1d1696784..234689aa4 100644 --- a/bin/varnishtest/tests/m00008.vtc +++ b/bin/varnishtest/tests/m00008.vtc @@ -14,7 +14,10 @@ varnish v1 -errvcl {Symbol 'debug' has wrong type (sub), expected vmod:} { varnish v1 -errvcl {Another module already imported as foo} { import debug as foo; - import directors as foo; + import + directors + as + foo; } server s1 { From dridi.boukelmoune at gmail.com Fri May 28 11:55:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 11:55:06 +0000 (UTC) Subject: [master] 3f9ccf94d varnishtest: Give the tunnel enough space for IPv6 Message-ID: <20210528115506.847DA11ABEB@lists.varnish-cache.org> commit 3f9ccf94d795e7ea517af62fbc0acfe7362ae6bc Author: Dridi Boukelmoune Date: Fri May 28 13:53:33 2021 +0200 varnishtest: Give the tunnel enough space for IPv6 diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index 418ce9195..95e3b880e 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -136,8 +136,8 @@ struct tunnel { char listen[256]; int lsock; - char laddr[32]; - char lport[32]; + char laddr[VTCP_ADDRBUFSIZE]; + char lport[VTCP_PORTBUFSIZE]; int asock; @@ -593,8 +593,7 @@ tunnel_listen(struct tunnel *t) assert(t->lsock > 0); sua = VSA_getsockname(t->lsock, buf, sizeof buf); AN(sua); - VTCP_name(sua, t->laddr, sizeof t->laddr, - t->lport, sizeof t->lport); + VTCP_name(sua, t->laddr, sizeof t->laddr, t->lport, sizeof t->lport); /* Record the actual port, and reuse it on subsequent starts */ if (VSA_Get_Proto(sua) == AF_INET) From dridi.boukelmoune at gmail.com Fri May 28 12:20:07 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 12:20:07 +0000 (UTC) Subject: [master] 3ec95ae46 varnishtest: Retire unused field Message-ID: <20210528122007.17510494D@lists.varnish-cache.org> commit 3ec95ae464185a8d5d041079c307fc7413794115 Author: Dridi Boukelmoune Date: Fri May 28 14:18:41 2021 +0200 varnishtest: Retire unused field diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index 95e3b880e..7f1812973 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -132,7 +132,6 @@ struct tunnel { char connect[256]; int csock; - const char *caddr; char listen[256]; int lsock; From dridi.boukelmoune at gmail.com Fri May 28 12:39:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 12:39:05 +0000 (UTC) Subject: [master] d4cf677ae vtc: Stabilize a21 Message-ID: <20210528123905.251AE5467@lists.varnish-cache.org> commit d4cf677ae48a46317bab432104c3729913d7a9db Author: Dridi Boukelmoune Date: Fri May 28 14:34:59 2021 +0200 vtc: Stabilize a21 And make a note that we should avoid pthread_cancel(3) in the future. diff --git a/bin/varnishtest/tests/a00021.vtc b/bin/varnishtest/tests/a00021.vtc index 9eb158582..43688b0f8 100644 --- a/bin/varnishtest/tests/a00021.vtc +++ b/bin/varnishtest/tests/a00021.vtc @@ -44,3 +44,5 @@ client c2 -connect "${t2_sock}" { txreq rxresp } -run + +tunnel t2 -wait From dridi.boukelmoune at gmail.com Fri May 28 13:24:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 13:24:06 +0000 (UTC) Subject: [master] 1cee4d37b varnishtest: Address flexelint's tunnel complaints Message-ID: <20210528132406.271366BD7@lists.varnish-cache.org> commit 1cee4d37bd68fbf47c4db595e5c4bdf10f602f0a Author: Dridi Boukelmoune Date: Fri May 28 15:22:54 2021 +0200 varnishtest: Address flexelint's tunnel complaints diff --git a/bin/varnishtest/vtc.h b/bin/varnishtest/vtc.h index d2b449359..5380f74cd 100644 --- a/bin/varnishtest/vtc.h +++ b/bin/varnishtest/vtc.h @@ -148,8 +148,6 @@ void start_h2(struct http *hp); void stop_h2(struct http *hp); void b64_settings(const struct http *hp, const char *s); -cmd_f cmd_tunnel; - /* vtc_gzip.c */ void vtc_gzip(struct http *, const char *, char **, long *); void vtc_gunzip(struct http *, char *, long *); diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index 7f1812973..7c579cf27 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -290,7 +290,7 @@ cmd_tunnel_resume(CMD_ARGS) AZ(pthread_mutex_unlock(&t->mtx)); } -const struct cmds tunnel_cmds[] = { +static const struct cmds tunnel_cmds[] = { #define CMD_TUNNEL(n) { #n, cmd_tunnel_##n }, CMD_TUNNEL(pause) CMD_TUNNEL(send) @@ -305,7 +305,7 @@ const struct cmds tunnel_cmds[] = { */ static void -tunnel_read(struct tunnel *t, struct vtclog *vl, struct pollfd *pfd, +tunnel_read(struct tunnel *t, struct vtclog *vl, const struct pollfd *pfd, struct tunnel_lane *lane) { size_t len; From phk at FreeBSD.org Fri May 28 13:26:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 28 May 2021 13:26:05 +0000 (UTC) Subject: [master] 5ef563a74 Convert CNUM and FNUM to double as part of lexing Message-ID: <20210528132605.B155A6E7A@lists.varnish-cache.org> commit 5ef563a742fb73586514bac6008625c7a521f68f Author: Poul-Henning Kamp Date: Fri May 28 13:14:41 2021 +0000 Convert CNUM and FNUM to double as part of lexing diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h index f97e5b46d..517dab5a6 100644 --- a/lib/libvcc/vcc_compile.h +++ b/lib/libvcc/vcc_compile.h @@ -101,6 +101,7 @@ struct token { VTAILQ_ENTRY(token) src_list; unsigned cnt; char *dec; + double num; }; /*---------------------------------------------------------------------*/ diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 020ada098..7d8d82926 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -691,13 +691,13 @@ vcc_number(struct vcc *tl, struct expr **e, vcc_type_t fmt, const char *sign) t = tl->t; vcc_NextToken(tl); if (tl->t->tok == ID) { - e1 = vcc_mk_expr(DURATION, "(%s%.*s) * %g", - sign, PF(t), vcc_DurationUnit(tl)); + e1 = vcc_mk_expr(DURATION, "%s%.3f * %g", + sign, t->num, vcc_DurationUnit(tl)); ERRCHK(tl); } else if (fmt == REAL || t->tok == FNUM) { - e1 = vcc_mk_expr(REAL, "%s%.*s", sign, PF(t)); + e1 = vcc_mk_expr(REAL, "%s%.3f", sign, t->num); } else { - e1 = vcc_mk_expr(INT, "%s%.*s", sign, PF(t)); + e1 = vcc_mk_expr(INT, "%s%.0f", sign, t->num); } } e1->constant = EXPR_CONST; diff --git a/lib/libvcc/vcc_token.c b/lib/libvcc/vcc_token.c index 05f8d3627..35cc6421b 100644 --- a/lib/libvcc/vcc_token.c +++ b/lib/libvcc/vcc_token.c @@ -446,6 +446,7 @@ static const char * vcc_lex_number(struct vcc *tl, struct source *sp, const char *p) { const char *q, *r; + char *s; for (q = p; q < sp->e; q++) if (!vct_isdigit(*q)) @@ -457,6 +458,8 @@ vcc_lex_number(struct vcc *tl, struct source *sp, const char *p) vcc_ErrWhere(tl, tl->t); return (NULL); } + tl->t->num = strtod(p, &s); + assert(s == tl->t->e); return (q); } r = ++q; @@ -469,6 +472,8 @@ vcc_lex_number(struct vcc *tl, struct source *sp, const char *p) vcc_ErrWhere(tl, tl->t); return(NULL); } + tl->t->num = strtod(p, &s); + assert(s == tl->t->e); return (r); } diff --git a/lib/libvcc/vcc_utils.c b/lib/libvcc/vcc_utils.c index 47f6ab00c..7ced4edc1 100644 --- a/lib/libvcc/vcc_utils.c +++ b/lib/libvcc/vcc_utils.c @@ -284,20 +284,13 @@ vcc_DurationUnit(struct vcc *tl) uint64_t vcc_UintVal(struct vcc *tl) { - const char *p, *errtxt; int64_t retval; if (tl->t->tok != CNUM) { Expect(tl, CNUM); return (0); } - p = tl->t->b; - retval = SF_Parse_Integer(&p, &errtxt); - if (errno) { - VSB_printf(tl->sb, "Bad UINT: %s\n", errtxt); - vcc_ErrWhere(tl, tl->t); - return (0); - } + retval = (int64_t)round(tl->t->num); if (retval < 0) { VSB_printf(tl->sb, "UINT cannot be negative\n"); vcc_ErrWhere(tl, tl->t); @@ -310,20 +303,13 @@ vcc_UintVal(struct vcc *tl) static double vcc_DoubleVal(struct vcc *tl) { - const char *p, *errtxt; double retval; if (tl->t->tok != CNUM && tl->t->tok != FNUM) { Expect(tl, CNUM); return (0); } - p = tl->t->b; - retval = SF_Parse_Decimal(&p, &errtxt); - if (errno) { - VSB_printf(tl->sb, "Bad REAL: %s\n", errtxt); - vcc_ErrWhere(tl, tl->t); - return (0); - } + retval = tl->t->num; vcc_NextToken(tl); return (retval); } @@ -349,19 +335,13 @@ vcc_ByteVal(struct vcc *tl, VCL_INT *d) { double v; VCL_INT retval; - const char *p, *errtxt; + const char *errtxt; if (tl->t->tok != CNUM && tl->t->tok != FNUM) { Expect(tl, CNUM); return; } - p = tl->t->b; - v = SF_Parse_Number(&p, &errtxt); - if (errno) { - VSB_printf(tl->sb, "Bad BYTES: %s\n", errtxt); - vcc_ErrWhere(tl, tl->t); - return; - } + v = tl->t->num; vcc_NextToken(tl); if (tl->t->tok != ID) { VSB_cat(tl->sb, "Expected BYTES unit (B, KB, MB...) got "); From phk at FreeBSD.org Fri May 28 13:26:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 28 May 2021 13:26:05 +0000 (UTC) Subject: [master] 6137fb6d9 Merge branch 'master' of github.com:varnishcache/varnish-cache Message-ID: <20210528132605.C630B6E7E@lists.varnish-cache.org> commit 6137fb6d9e5e797eaaa61355a73fc962a3ca1068 Merge: 5ef563a74 1cee4d37b Author: Poul-Henning Kamp Date: Fri May 28 13:25:38 2021 +0000 Merge branch 'master' of github.com:varnishcache/varnish-cache From dridi.boukelmoune at gmail.com Fri May 28 13:45:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 13:45:06 +0000 (UTC) Subject: [master] a52a1dd47 varnishtest: One more flexelint complaint Message-ID: <20210528134506.955897B10@lists.varnish-cache.org> commit a52a1dd47e98da53ecb913c9faf2a3de26444136 Author: Dridi Boukelmoune Date: Fri May 28 15:38:52 2021 +0200 varnishtest: One more flexelint complaint The assertion above wasn't enough apparently. diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index 7c579cf27..8e1f9e595 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -362,7 +362,7 @@ tunnel_write(struct tunnel *t, struct vtclog *vl, struct tunnel_lane *lane, AZ(pthread_mutex_lock(&t->mtx)); if (lane->wrk_len > 0 && lane->buf_len != -1) { assert(lane->buf_len >= 0); - assert(lane->wrk_len >= lane->buf_len); + assert(lane->wrk_len >= (size_t)lane->buf_len); lane->wrk_len -= lane->buf_len; } lane->buf_len = l; From dridi.boukelmoune at gmail.com Fri May 28 14:21:05 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 28 May 2021 14:21:05 +0000 (UTC) Subject: [master] f00f23d8f varnishtest: Don't cancel tunnel threads Message-ID: <20210528142105.CA23B9D9E@lists.varnish-cache.org> commit f00f23d8f892a4b05e84d152e109e3da86a4d786 Author: Dridi Boukelmoune Date: Fri May 28 16:16:29 2021 +0200 varnishtest: Don't cancel tunnel threads Unlike other threads we cancel in varnishtest there's 2 per tunnel and they need coordination between each other. The spec thread remains as-is and the poll thread now checks vtc_stop after each condwait. With this I'm no longer able to reproduce a21's timeout. Reverts d4cf677ae48a46317bab432104c3729913d7a9db diff --git a/bin/varnishtest/tests/a00021.vtc b/bin/varnishtest/tests/a00021.vtc index 43688b0f8..9eb158582 100644 --- a/bin/varnishtest/tests/a00021.vtc +++ b/bin/varnishtest/tests/a00021.vtc @@ -44,5 +44,3 @@ client c2 -connect "${t2_sock}" { txreq rxresp } -run - -tunnel t2 -wait diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index 8e1f9e595..d26c73e16 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -386,11 +386,14 @@ tunnel_poll_thread(void *priv) while (tunnel_is_open(t) && !vtc_stop) { AZ(pthread_mutex_lock(&t->mtx)); /* NB: can be woken up by `tunnel tX -wait` */ - while (t->state == TUNNEL_ACCEPT) + while (t->state == TUNNEL_ACCEPT && !vtc_stop) AZ(pthread_cond_wait(&t->cond, &t->mtx)); state = t->state; AZ(pthread_mutex_unlock(&t->mtx)); + if (vtc_stop) + break; + assert(state < TUNNEL_POLL_DONE); memset(pfd, 0, sizeof pfd); @@ -415,6 +418,9 @@ tunnel_poll_thread(void *priv) } AZ(pthread_mutex_unlock(&t->mtx)); + if (vtc_stop) + break; + tunnel_write(t, vl, t->send_lane, "Sending"); tunnel_write(t, vl, t->recv_lane, "Receiving"); } @@ -651,14 +657,10 @@ tunnel_wait(struct tunnel *t) AZ(pthread_cond_signal(&t->cond)); AZ(pthread_join(t->tspec, &res)); - if (res == PTHREAD_CANCELED && !vtc_stop) - vtc_fatal(t->vl, "Tunnel spec canceled"); if (res != NULL && !vtc_stop) vtc_fatal(t->vl, "Tunnel spec returned \"%p\"", res); AZ(pthread_join(t->tpoll, &res)); - if (res == PTHREAD_CANCELED && !vtc_stop) - vtc_fatal(t->vl, "Tunnel poll canceled"); if (res != NULL && !vtc_stop) vtc_fatal(t->vl, "Tunnel poll returned \"%p\"", res); @@ -679,7 +681,6 @@ static void tunnel_reset(void) { struct tunnel *t; - enum tunnel_state_e state; while (1) { AZ(pthread_mutex_lock(&tunnel_mtx)); @@ -691,15 +692,7 @@ tunnel_reset(void) if (t == NULL) break; - AZ(pthread_mutex_lock(&t->mtx)); - state = t->state; - if (state < TUNNEL_POLL_DONE) - (void)pthread_cancel(t->tpoll); - if (state < TUNNEL_SPEC_DONE) - (void)pthread_cancel(t->tspec); - AZ(pthread_mutex_unlock(&t->mtx)); - - if (state != TUNNEL_STOPPED) + if (t->state != TUNNEL_STOPPED) tunnel_wait(t); tunnel_delete(t); } From dridi.boukelmoune at gmail.com Mon May 31 04:38:06 2021 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 31 May 2021 04:38:06 +0000 (UTC) Subject: [master] 85e79d069 varnishtest: Remove critical section Message-ID: <20210531043807.00A671176FB@lists.varnish-cache.org> commit 85e79d069ff37da1154936e53eafd5ce7c18c2f6 Author: Dridi Boukelmoune Date: Mon May 31 06:32:36 2021 +0200 varnishtest: Remove critical section The buf_len field is only ever modified by the poll thread and should not need to be guarded here. This will hopefully silence a coverity warning: CID 1485138: Concurrent data access violations (ATOMICITY) Using an unreliable value of "l" inside the second locked section. If the data that "l" depends on was changed by another thread, this use might be incorrect. It might on the other hand complain even harder, I have an alternative if that happens. diff --git a/bin/varnishtest/vtc_tunnel.c b/bin/varnishtest/vtc_tunnel.c index d26c73e16..28fc05ec0 100644 --- a/bin/varnishtest/vtc_tunnel.c +++ b/bin/varnishtest/vtc_tunnel.c @@ -345,9 +345,7 @@ tunnel_write(struct tunnel *t, struct vtclog *vl, struct tunnel_lane *lane, ssize_t res, l; p = lane->buf; - AZ(pthread_mutex_lock(&t->mtx)); l = lane->buf_len; - AZ(pthread_mutex_unlock(&t->mtx)); if (l > 0) vtc_log(vl, 3, "%s %zd bytes", action, l); From phk at FreeBSD.org Mon May 31 10:51:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 10:51:06 +0000 (UTC) Subject: [master] 68684fec9 Use VCL_BYTES instead of int64_t return type for VRT_stevedore_*() Message-ID: <20210531105106.362CD96B3@lists.varnish-cache.org> commit 68684fec9f52afbf47c8ed72b1086951ae1ee9f1 Author: Poul-Henning Kamp Date: Mon May 31 10:49:56 2021 +0000 Use VCL_BYTES instead of int64_t return type for VRT_stevedore_*() diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py index 1b23b6282..c1117905a 100755 --- a/lib/libvcc/generate.py +++ b/lib/libvcc/generate.py @@ -760,7 +760,7 @@ parse_var_doc(join(srcroot, "doc/sphinx/reference/vcl_var.rst")) fo.write("}\n") for i in stv_variables: - fh.write(vcltypes[i[1]].c + " VRT_stevedore_" + i[0] + "(VCL_STEVEDORE);\n") + fh.write("VCL_" + i[1] + " VRT_stevedore_" + i[0] + "(VCL_STEVEDORE);\n") fo.write("\n/* VCL type identifiers */\n") From phk at FreeBSD.org Mon May 31 10:51:06 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 10:51:06 +0000 (UTC) Subject: [master] 747f35b7f Use VCL_BYTES for size-accounting. Message-ID: <20210531105106.5409D96B7@lists.varnish-cache.org> commit 747f35b7f04d7fc85de92a20e5dbe250b007b0a4 Author: Poul-Henning Kamp Date: Mon May 31 10:50:27 2021 +0000 Use VCL_BYTES for size-accounting. diff --git a/bin/varnishd/storage/storage_malloc.c b/bin/varnishd/storage/storage_malloc.c index ec99c1738..93dd52747 100644 --- a/bin/varnishd/storage/storage_malloc.c +++ b/bin/varnishd/storage/storage_malloc.c @@ -50,8 +50,8 @@ struct sma_sc { unsigned magic; #define SMA_SC_MAGIC 0x1ac8a345 struct lock sma_mtx; - size_t sma_max; - size_t sma_alloc; + VCL_BYTES sma_max; + VCL_BYTES sma_alloc; struct VSC_sma *stats; }; @@ -83,7 +83,7 @@ sma_alloc(const struct stevedore *st, size_t size) sma_sc->stats->c_bytes += size; sma_sc->stats->g_alloc++; sma_sc->stats->g_bytes += size; - if (sma_sc->sma_max != SIZE_MAX) + if (sma_sc->sma_max != VRT_INTEGER_MAX) sma_sc->stats->g_space -= size; } Lck_Unlock(&sma_sc->sma_mtx); @@ -117,7 +117,7 @@ sma_alloc(const struct stevedore *st, size_t size) sma_sc->stats->c_bytes -= size; sma_sc->stats->g_alloc--; sma_sc->stats->g_bytes -= size; - if (sma_sc->sma_max != SIZE_MAX) + if (sma_sc->sma_max != VRT_INTEGER_MAX) sma_sc->stats->g_space += size; Lck_Unlock(&sma_sc->sma_mtx); return (NULL); @@ -146,7 +146,7 @@ sma_free(struct storage *s) sma_sc->stats->g_alloc--; sma_sc->stats->g_bytes -= sma->sz; sma_sc->stats->c_freed += sma->sz; - if (sma_sc->sma_max != SIZE_MAX) + if (sma_sc->sma_max != VRT_INTEGER_MAX) sma_sc->stats->g_space += sma->sz; Lck_Unlock(&sma_sc->sma_mtx); free(sma->s.ptr); @@ -181,8 +181,8 @@ sma_init(struct stevedore *parent, int ac, char * const *av) ASSERT_MGT(); ALLOC_OBJ(sc, SMA_SC_MAGIC); AN(sc); - sc->sma_max = SIZE_MAX; - assert(sc->sma_max == SIZE_MAX); + sc->sma_max = VRT_INTEGER_MAX; + assert(sc->sma_max == VRT_INTEGER_MAX); parent->priv = sc; AZ(av[ac]); @@ -216,7 +216,7 @@ sma_open(struct stevedore *st) CAST_OBJ_NOTNULL(sma_sc, st->priv, SMA_SC_MAGIC); Lck_New(&sma_sc->sma_mtx, lck_sma); sma_sc->stats = VSC_sma_New(NULL, NULL, st->ident); - if (sma_sc->sma_max != SIZE_MAX) + if (sma_sc->sma_max != VRT_INTEGER_MAX) sma_sc->stats->g_space = sma_sc->sma_max; } From phk at FreeBSD.org Mon May 31 17:13:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 17:13:05 +0000 (UTC) Subject: [master] 416c8d457 Preemptively reduce to legal number of decimals Message-ID: <20210531171305.1C1BB10512D@lists.varnish-cache.org> commit 416c8d457be950c98970c36920c25233b91e0ad5 Author: Poul-Henning Kamp Date: Mon May 31 14:43:59 2021 +0000 Preemptively reduce to legal number of decimals diff --git a/bin/varnishtest/tests/m00005.vtc b/bin/varnishtest/tests/m00005.vtc index 970e26e8e..de5f7c4d7 100644 --- a/bin/varnishtest/tests/m00005.vtc +++ b/bin/varnishtest/tests/m00005.vtc @@ -113,7 +113,7 @@ client c1 { # --- - txreq -hdr "d-real: 0.010" -hdr "b-real: 2048.99999" + txreq -hdr "d-real: 0.010" -hdr "b-real: 2048.999" rxresp expect resp.http.duration == 1000000.010 expect resp.http.bytes == 2560 From phk at FreeBSD.org Mon May 31 17:13:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 17:13:05 +0000 (UTC) Subject: [master] 3cd72fc67 Test both as string and real argument Message-ID: <20210531171305.3003D105130@lists.varnish-cache.org> commit 3cd72fc678f81474c237710fbaf096cf0fe640e1 Author: Poul-Henning Kamp Date: Mon May 31 14:44:30 2021 +0000 Test both as string and real argument diff --git a/bin/varnishtest/tests/m00016.vtc b/bin/varnishtest/tests/m00016.vtc index 04b58f236..47ee311fe 100644 --- a/bin/varnishtest/tests/m00016.vtc +++ b/bin/varnishtest/tests/m00016.vtc @@ -19,8 +19,11 @@ varnish v1 -vcl+backend { set resp.http.x-qux = std.real2integer( std.real(req.http.foo, 2.0), 2); - set resp.http.x-xyzzy = std.integer( - std.real(req.http.foo, 2.0), 2); + set resp.http.x-xyzzy1 = std.integer( + s=std.real(req.http.foo, 2.0)); + + set resp.http.x-xyzzy2 = std.integer( + real=std.real(req.http.foo, 2.0)); set resp.http.x-int-fallback = std.real2integer(123456789012.345 * 1000.0 * 10, 2); } @@ -32,7 +35,8 @@ client c1 { expect resp.http.x-foo == resp.http.x-bar expect resp.http.x-baz == 1140618699.000 expect resp.http.x-qux == 1140618699 - expect resp.http.x-xyzzy == resp.http.x-qux + expect resp.http.x-xyzzy1 == resp.http.x-qux + expect resp.http.x-xyzzy2 == resp.http.x-qux expect resp.http.x-int-fallback == 2 } -run From phk at FreeBSD.org Mon May 31 17:13:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 17:13:05 +0000 (UTC) Subject: [master] 5e94577f7 Expect 503s Message-ID: <20210531171305.4D329105133@lists.varnish-cache.org> commit 5e94577f7353bbb371834e839147b63aee1965bc Author: Poul-Henning Kamp Date: Mon May 31 14:57:50 2021 +0000 Expect 503s diff --git a/bin/varnishtest/tests/i00001.vtc b/bin/varnishtest/tests/i00001.vtc index a56300eb7..48f46e9ac 100644 --- a/bin/varnishtest/tests/i00001.vtc +++ b/bin/varnishtest/tests/i00001.vtc @@ -45,11 +45,13 @@ logexpect l1 -v v1 -g raw { client c1 { txreq -hdr "foo: 1" rxresp + expect resp.status == 503 } -run client c1 { txreq -hdr "bar: 1" rxresp + expect resp.status == 503 } -run logexpect l1 -wait From phk at FreeBSD.org Mon May 31 17:13:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 17:13:05 +0000 (UTC) Subject: [master] de19176b1 Make sure to not involve 64 bit overflow Message-ID: <20210531171305.68EE1105138@lists.varnish-cache.org> commit de19176b1d67c6b173ca6f604dcf2978dd2da891 Author: Poul-Henning Kamp Date: Mon May 31 17:05:33 2021 +0000 Make sure to not involve 64 bit overflow diff --git a/bin/varnishtest/tests/i00001.vtc b/bin/varnishtest/tests/i00001.vtc index 48f46e9ac..b952a5016 100644 --- a/bin/varnishtest/tests/i00001.vtc +++ b/bin/varnishtest/tests/i00001.vtc @@ -32,7 +32,8 @@ varnish v1 -vcl+backend { set resp.http.foo = obj.ttl * 10000000000; } if (req.http.bar) { - set resp.http.bar = storage.Transient.free_space * 10000000 * 100000000; + # unlimited malloc stevedore returns VCL_INT_MAX - epsilon + set resp.http.bar = storage.Transient.free_space * 10; } } } -start From phk at FreeBSD.org Mon May 31 17:13:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 17:13:05 +0000 (UTC) Subject: [master] b3799d224 Add a cast for FlexeLint Message-ID: <20210531171305.8432110513C@lists.varnish-cache.org> commit b3799d22442ce341ac7d2a8991644f760bd8ebf0 Author: Poul-Henning Kamp Date: Mon May 31 17:06:44 2021 +0000 Add a cast for FlexeLint diff --git a/bin/varnishd/storage/storage_malloc.c b/bin/varnishd/storage/storage_malloc.c index 93dd52747..013d5e58d 100644 --- a/bin/varnishd/storage/storage_malloc.c +++ b/bin/varnishd/storage/storage_malloc.c @@ -75,7 +75,7 @@ sma_alloc(const struct stevedore *st, size_t size) CAST_OBJ_NOTNULL(sma_sc, st->priv, SMA_SC_MAGIC); Lck_Lock(&sma_sc->sma_mtx); sma_sc->stats->c_req++; - if (sma_sc->sma_alloc + size > sma_sc->sma_max) { + if (sma_sc->sma_alloc + (VCL_BYTES)size > sma_sc->sma_max) { sma_sc->stats->c_fail++; size = 0; } else { From phk at FreeBSD.org Mon May 31 17:13:05 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 17:13:05 +0000 (UTC) Subject: [master] d69149c7a Make SF_Parse_(numeric) ignore leading and trailing ows. Message-ID: <20210531171305.9DA7A105140@lists.varnish-cache.org> commit d69149c7a0c441fcc8c8c9f35093dbd396888bfc Author: Poul-Henning Kamp Date: Mon May 31 17:11:28 2021 +0000 Make SF_Parse_(numeric) ignore leading and trailing ows. diff --git a/lib/libvarnish/vnum.c b/lib/libvarnish/vnum.c index 2cf665480..ce66fc7e1 100644 --- a/lib/libvarnish/vnum.c +++ b/lib/libvarnish/vnum.c @@ -81,6 +81,8 @@ sf_parse_int(const char **ipp, const char **errtxt, int maxdig) retval *= 10; retval += *(*ipp)++ - 0x30; } + while (vct_isows(*(*ipp))) + (*ipp)++; if (ndig == 0) BAIL(negative ? err_invalid_num : err_miss_num); if (negative) @@ -98,48 +100,56 @@ SF_Parse_Integer(const char **ipp, const char **errtxt) double SF_Parse_Decimal(const char **ipp, const char **errtxt) { - double retval; + double retval, scale = 1; + int ndig; retval = (double)sf_parse_int(ipp, errtxt, 12); - if (*(*ipp) != '.') - return (retval); - (*ipp)++; - if (!vct_isdigit(*(*ipp))) - return (retval); - retval += .1 * (*(*ipp)++ - 0x30); - if (!vct_isdigit(*(*ipp))) - return (retval); - retval += .01 * (*(*ipp)++ - 0x30); - if (!vct_isdigit(*(*ipp))) - return (retval); - retval += .001 * (*(*ipp)++ - 0x30); - if (vct_isdigit(*(*ipp))) - BAIL(err_fatnum); + if (retval < 0) + scale = -scale; + do { + if (*(*ipp) != '.') + break; + (*ipp)++; + for(ndig = 0; ndig < 3; ndig++) { + scale *= .1; + if (!vct_isdigit(*(*ipp))) + break; + retval += scale * (*(*ipp)++ - 0x30); + } + if (vct_isdigit(*(*ipp))) + BAIL(err_fatnum); + } while (0); + while (vct_isows(*(*ipp))) + (*ipp)++; return (retval); } double SF_Parse_Number(const char **ipp, const char **errtxt) { - double retval; + double retval, scale = 1; + int ndig; retval = (double)sf_parse_int(ipp, errtxt, 15); - if (*(*ipp) != '.') - return (retval); - if (retval < -999999999999 || retval > 999999999999) - BAIL(err_fatnum); - (*ipp)++; - if (!vct_isdigit(*(*ipp))) - return (retval); - retval += .1 * (*(*ipp)++ - 0x30); - if (!vct_isdigit(*(*ipp))) - return (retval); - retval += .01 * (*(*ipp)++ - 0x30); - if (!vct_isdigit(*(*ipp))) - return (retval); - retval += .001 * (*(*ipp)++ - 0x30); - if (vct_isdigit(*(*ipp))) - BAIL(err_fatnum); + if (retval < 0) + scale = -scale; + do { + if (*(*ipp) != '.') + break; + if (retval < -999999999999 || retval > 999999999999) + BAIL(err_fatnum); + (*ipp)++; + for(ndig = 0; ndig < 3; ndig++) { + scale *= .1; + if (!vct_isdigit(*(*ipp))) + break; + retval += scale * (*(*ipp)++ - 0x30); + } + if (vct_isdigit(*(*ipp))) + BAIL(err_fatnum); + } while (0); + while (vct_isows(*(*ipp))) + (*ipp)++; return (retval); } From phk at FreeBSD.org Mon May 31 19:16:04 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 19:16:04 +0000 (UTC) Subject: [master] 22881138e Blindly attempt to fix space accounting in storage_umem.c (analog of storage_malloc.c) Message-ID: <20210531191604.F25BA1089FB@lists.varnish-cache.org> commit 22881138edf748f82a44cf14f5dac2648b476709 Author: Poul-Henning Kamp Date: Mon May 31 19:15:17 2021 +0000 Blindly attempt to fix space accounting in storage_umem.c (analog of storage_malloc.c) diff --git a/bin/varnishd/storage/storage_umem.c b/bin/varnishd/storage/storage_umem.c index bc3458e9c..3da22a35e 100644 --- a/bin/varnishd/storage/storage_umem.c +++ b/bin/varnishd/storage/storage_umem.c @@ -58,8 +58,8 @@ struct smu_sc { unsigned magic; #define SMU_SC_MAGIC 0x7695f68e struct lock smu_mtx; - size_t smu_max; - size_t smu_alloc; + VCL_INT smu_max; + VCL_INT smu_alloc; struct VSC_smu *stats; umem_cache_t *smu_cache; }; @@ -151,7 +151,7 @@ smu_alloc(const struct stevedore *st, size_t size) CAST_OBJ_NOTNULL(smu_sc, st->priv, SMU_SC_MAGIC); Lck_Lock(&smu_sc->smu_mtx); smu_sc->stats->c_req++; - if (smu_sc->smu_alloc + size > smu_sc->smu_max) { + if (smu_sc->smu_alloc + (int64_t)size > smu_sc->smu_max) { smu_sc->stats->c_fail++; size = 0; } else { @@ -159,7 +159,7 @@ smu_alloc(const struct stevedore *st, size_t size) smu_sc->stats->c_bytes += size; smu_sc->stats->g_alloc++; smu_sc->stats->g_bytes += size; - if (smu_sc->smu_max != SIZE_MAX) + if (smu_sc->smu_max != VRT_INTEGER_MAX) smu_sc->stats->g_space -= size; } Lck_Unlock(&smu_sc->smu_mtx); @@ -194,7 +194,7 @@ smu_alloc(const struct stevedore *st, size_t size) smu_sc->stats->c_bytes -= size; smu_sc->stats->g_alloc--; smu_sc->stats->g_bytes -= size; - if (smu_sc->smu_max != SIZE_MAX) + if (smu_sc->smu_max != VRT_INTEGER_MAX) smu_sc->stats->g_space += size; Lck_Unlock(&smu_sc->smu_mtx); return (NULL); @@ -224,7 +224,7 @@ smu_free(struct storage *s) sc->stats->g_alloc--; sc->stats->g_bytes -= smu->sz; sc->stats->c_freed += smu->sz; - if (sc->smu_max != SIZE_MAX) + if (sc->smu_max != VRT_INTEGER_MAX) sc->stats->g_space += smu->sz; Lck_Unlock(&sc->smu_mtx); @@ -300,8 +300,8 @@ smu_init(struct stevedore *parent, int ac, char * const *av) ASSERT_MGT(); ALLOC_OBJ(sc, SMU_SC_MAGIC); AN(sc); - sc->smu_max = SIZE_MAX; - assert(sc->smu_max == SIZE_MAX); + sc->smu_max = VRT_INTEGER_MAX; + assert(sc->smu_max == VRT_INTEGER_MAX); parent->priv = sc; AZ(av[ac]); @@ -409,7 +409,7 @@ smu_open(struct stevedore *st) CAST_OBJ_NOTNULL(smu_sc, st->priv, SMU_SC_MAGIC); Lck_New(&smu_sc->smu_mtx, lck_smu); smu_sc->stats = VSC_smu_New(NULL, NULL, st->ident); - if (smu_sc->smu_max != SIZE_MAX) + if (smu_sc->smu_max != VRT_INTEGER_MAX) smu_sc->stats->g_space = smu_sc->smu_max; smu_open_init(); From dridi at varni.sh Mon May 31 19:50:08 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 31 May 2021 19:50:08 +0000 Subject: [master] 22881138e Blindly attempt to fix space accounting in storage_umem.c (analog of storage_malloc.c) In-Reply-To: <20210531191604.F25BA1089FB@lists.varnish-cache.org> References: <20210531191604.F25BA1089FB@lists.varnish-cache.org> Message-ID: On Mon, May 31, 2021 at 7:16 PM Poul-Henning Kamp wrote: > > > commit 22881138edf748f82a44cf14f5dac2648b476709 > Author: Poul-Henning Kamp > Date: Mon May 31 19:15:17 2021 +0000 > > Blindly attempt to fix space accounting in storage_umem.c > (analog of storage_malloc.c) > > diff --git a/bin/varnishd/storage/storage_umem.c b/bin/varnishd/storage/storage_umem.c > index bc3458e9c..3da22a35e 100644 > --- a/bin/varnishd/storage/storage_umem.c > +++ b/bin/varnishd/storage/storage_umem.c > @@ -58,8 +58,8 @@ struct smu_sc { > unsigned magic; > #define SMU_SC_MAGIC 0x7695f68e > struct lock smu_mtx; > - size_t smu_max; > - size_t smu_alloc; > + VCL_INT smu_max; > + VCL_INT smu_alloc; Did you mean to use VCL_BYTES like in struct sma_sc? From phk at phk.freebsd.dk Mon May 31 22:11:38 2021 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 31 May 2021 22:11:38 +0000 Subject: [master] 22881138e Blindly attempt to fix space accounting in storage_umem.c (analog of storage_malloc.c) In-Reply-To: References: <20210531191604.F25BA1089FB@lists.varnish-cache.org> Message-ID: <88549.1622499098@critter.freebsd.dk> -------- Dridi Boukelmoune writes: > On Mon, May 31, 2021 at 7:16 PM Poul-Henning Kamp wrote: > > > > > > commit 22881138edf748f82a44cf14f5dac2648b476709 > > Author: Poul-Henning Kamp > > Date: Mon May 31 19:15:17 2021 +0000 > > > > Blindly attempt to fix space accounting in storage_umem.c > > (analog of storage_malloc.c) > > > > diff --git a/bin/varnishd/storage/storage_umem.c b/bin/varnishd/storage/sto > rage_umem.c > > index bc3458e9c..3da22a35e 100644 > > --- a/bin/varnishd/storage/storage_umem.c > > +++ b/bin/varnishd/storage/storage_umem.c > > @@ -58,8 +58,8 @@ struct smu_sc { > > unsigned magic; > > #define SMU_SC_MAGIC 0x7695f68e > > struct lock smu_mtx; > > - size_t smu_max; > > - size_t smu_alloc; > > + VCL_INT smu_max; > > + VCL_INT smu_alloc; > > Did you mean to use VCL_BYTES like in struct sma_sc? Indeed I did, thanks :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at FreeBSD.org Mon May 31 22:12:09 2021 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 31 May 2021 22:12:09 +0000 (UTC) Subject: [master] c42570327 VCL_BYTES, not VCL_INT. Message-ID: <20210531221209.C815610F71E@lists.varnish-cache.org> commit c42570327f1e4f746715179d20ad17e032647436 Author: Poul-Henning Kamp Date: Mon May 31 22:10:48 2021 +0000 VCL_BYTES, not VCL_INT. Spotted by: @dridi diff --git a/bin/varnishd/storage/storage_umem.c b/bin/varnishd/storage/storage_umem.c index 3da22a35e..d4fc52a0d 100644 --- a/bin/varnishd/storage/storage_umem.c +++ b/bin/varnishd/storage/storage_umem.c @@ -58,8 +58,8 @@ struct smu_sc { unsigned magic; #define SMU_SC_MAGIC 0x7695f68e struct lock smu_mtx; - VCL_INT smu_max; - VCL_INT smu_alloc; + VCL_BYTES smu_max; + VCL_BYTES smu_alloc; struct VSC_smu *stats; umem_cache_t *smu_cache; };