Dates

[Anders Berg]

On Jul 23, 2006, at 0:15, Poul-Henning Kamp wrote:

> In message <ujrbqrhfi16.fsf at cat.linpro.no>, Dag-Erling =?iso-8859-1? 
> Q?Sm=F8rgra
> v?= writes:
>
>>> We have passed this stage, and it is still not sane to release a  
>>> Alpha.
>>
>> I agree.  The code is a lot more solid in many areas than I expected
>> it to be at this stage, but there are still white spots on the map.
>
> Here is a brain-dump status report in no particular order.
>
> It should probably be in the wiki somewhere...
>
> ---------------------------------------------------------------------- 
> -
> What:
> 	Error handling on backend connections
> Situation:
> 	Right now we more or less panic on any sign of trouble.
> What needs to be done:
> 	We need to decide how to tie this into VCL
> 	Errors must be handled, presumably by retry.
> 	Timeouts probably desirable as well.
> Severity:
> 	alpha:	can live with
> 	beta:	must be improved
> 	1.0:	backend trouble SHALL not kill varnish
>

Agree.

I guess this is also tied to how Varnish handles "error/fault" on  
backend as well.


> ---------------------------------------------------------------------- 
> -
> What:
> 	Move more policy to VCL
> Situation:
> 	There seems to be no reason why we can't move more policy
> 	from C code to VCL code.
> What needs to be done:
> 	Make sure VCL support the necessary operations
> 	Move policy to default VCL functions.
> Severity:
> 	alpha:	can live with.
> 	beta:	can live with.
> 	1.0:	can live with.

Agree.

> ---------------------------------------------------------------------- 
> -
> What:
> 	VCL error implementation
> Situation:
> 	The error keyword in VCL is not implemented
> What needs to be done:
> 	Find out if this is really the right strategy, it looks
> 	more and more dubious to me.
> 	Maybe a redirect to an error document loaded with storage_tar
> 	would be more useful.
> Severity:
> 	alpha:	can live with minimal improvement
> 	beta:	can live with minimal improvement
> 	1.0:	can live with minimal improvement

Not sure what this one means. Is this the error we return to client?  
Or is this the error from backend?

By that I mean:

if (something true){

	error;

}

or a

if (backend.error){
	
	do something;

}


> ---------------------------------------------------------------------- 
> -
> What:
> 	Vary header matching
> Situation:
> 	We don't do it.
> What needs to be done:
> 	Implement it.
> 	Now that we have the http munging facilities mostly
> 	in place this is possible.
> 	Test against www.cnn.com for instance.
> Severity:
> 	alpha:	can live with
> 	beta:	can live with
> 	1.0:	can live with

Agree

> ---------------------------------------------------------------------- 
> -
> What:
> 	Performance statistics
> Situation:
> 	We have some
> What needs to be done:
> 	We need to figure out what is relevant stats
> 	We need to collect it.
> 	We need to make (some of) it available in VCL
> Severity:
> 	alpha:	can live with
> 	beta:	improvement desirable
> 	1.0:	TBD

Agree.

Stats that should be visible in VCL would from the top of my mind be.

Hits on documents. (Not sure about if we can implement this, but I am  
thinking in the line of 20.000 hits on /index.html last 2 min and ttl  
+2m. It certainly requires us to keep stats on each document/URL or a  
top-hit-list, and that may not be possible.)
Backend responstime.
Client bandwith/hits.


> ---------------------------------------------------------------------- 
> -
> What:
> 	CLI management interface
> Situation:
> 	Only minimally functional.
> What needs to be done:
> 	Reevaluate the list we created initially (ie: no unlisten)
> 	Implement TCP / SSH carried CLI port
> Severity:
> 	alpha:	can live with
> 	beta:	improvement desirable
> 	1.0:	TCP/SSH mandatory

"Reevaluate the list we created initially (ie: no unlisten" ?
Agree on the rest.

> ---------------------------------------------------------------------- 
> -
> What:
> 	VCL completeness
> Situation:
> 	VCL can't do much yet.
> What needs to be done:
> 	Go through our list, implement in proritized order
> Severity:
> 	alpha:	can live with
> 	beta:	HTTP header manipulation desirable
> 	1.0:	TBD

Agree.

Should I prioritize?


> ---------------------------------------------------------------------- 
> -
> What:
> 	VCL compiler error checking
> Situation:
> 	Errors from GCC and at load-time still happens.
> 	Failed loading of default/boot VCL is not currently fatal.
> What needs to be done:
> 	Check conditions better, for instance do getaddrinfo() on
> 	ACL targets and warn if non () entries fail.
> Severity:
> 	alpha:	can live with
> 	beta:	can live with
> 	1.0:	can live with
>

Agree.

We at least need getaddrinfo on the backend IP's, but I guess that  
this is meant on other cases?

> ---------------------------------------------------------------------- 
> -
> What:
> 	(regression-)testing
> Situation:
> 	We need a way to build regression tests.
> 	"varnishtester" is maybe a beginning.
> What needs to be done:
> 	1. (see above). 2. Think about problem 3. Write down code :-)
> Severity:
> 	alpha:	can live with
> 	beta:	would rather not live with
> 	1.0:	serious limitation

Agree.

The thing is that this would be a good tool for people writing code,  
to see that their implementation is correct. Helps debug alot I guess.


> ---------------------------------------------------------------------- 
> -
> What:
> 	Random memory corruption
> Situation:
> 	Most live test runs have seen one or two core dumps
> 	that look like pointer fandango after en hour or so.
> 	FlexeLint does not spot anything.
> What needs to be done:
> 	Run with debugging malloc of some kind, try to find.
> Severity:
> 	alpha:	can live with if no choice
> 	beta:	would really hate to live with
> 	1.0:	showstopper, we should be able to run 24h

Agree.

> ---------------------------------------------------------------------- 
> -
> What:
> 	cache process restart
> Situation:
> 	management process does not restart automatically
> 	(for debugging convenience)
> What needs to be done:
> 	management process should restart cache process.
> 	Bonus: save snapshot of shmem and get traceback
> 	from gdb as well.
> Severity:
> 	alpha:	can live with
> 	beta:	can live with
> 	1.0:	should have

Agree.

I liked the shmem snap. It would help debugging alot I guess.

> ---------------------------------------------------------------------- 
> -
> What:
> 	RFC compliance audit
> Situation:
> 	We don't know how many small details we are missing.
> What needs to be done:
> 	Go through RFC(s) and sourcecode and mark up RFC(s)
> 	with "not relevant", "deliberate variance", "not implemented"
> 	and "OK".  (OK preferably listing relevant testcases)
> Severity:
> 	alpha:	can live with
> 	beta:	can live with
> 	1.0:	can live with

Bump to must/should have on 1.0?

We are gonna need this sooner rather than later. Ppl are gonna ask  
for this as soon as they start using Varnish in a prod. envorionment  
is my guess.

> ---------------------------------------------------------------------- 
> -
> What:
> 	storage_file.c allocation strategy
> Situation:
> 	we have seen pessimisal allocation patterns with
> 	degenerate trafic patterns.
> 	Real World traffic looks basically OK, but no detailed
> 	analysis done.  Missing stats an issue.
> What needs to be done:
> 	Carefully consider mitigation once we have analyzed
> 	the issue.
> Severity:
> 	alpha:	no issue
> 	beta:	no issue
> 	1.0:	no issue

Agree.

1.0 should have a way to dump the stats, so users reporting an error  
in that area, can give us a clue as to what is happening.

> ---------------------------------------------------------------------- 
> -
> Nice to have:
> 	log-tailer which only shows entries which pertain to
> 	sessions from given IP# range.  (possibly generic
> 	log-tailer option ?)

Agree.

> ---------------------------------------------------------------------- 
> -
> Nice to have:
> 	"pigs" logtailer which shows top clients and their
> 	requests in usable format.

Agree.

> ---------------------------------------------------------------------- 
> -
> Nice to have:
> 	log-tailer which writes httpd format.

I wanna bump up this one. Ppl may need it for stats.

> ---------------------------------------------------------------------- 
> -
> Nice to have:
> 	Better stats presentation on screen

Agree.

> ---------------------------------------------------------------------- 
> -
> Nice to have:
> 	Allow purging through HTTP interface.
> 	Doesn't take much actually

Agree.

> ---------------------------------------------------------------------- 
> -
> Nice to have:
> 	TAR storage method for loading static contents.

Agree.


I know that bumps upwards are gonna resolve on more work, but a httpd  
format log is really a must-have I think. Not for VG, but for others.
I am currently looking into seeing if I can program any C at all :)  
httpd logg would be a good place to start I guess. A mail will follow  
after this one about the logging code.


Anders Berg


> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by  
> incompetence.
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev
>