Article delegate-en/3243 of [1-5169] on the server localhost:119
  upper oldest olders older1 this newer1 newers latest
[Top/Up] [oldest] - [Older+chunk] - [Newer+chunk] - [newest + Check]
Newsgroups: mail-lists.delegate-en

[DeleGate-En] Re: File size issue with FTP cache
24 May 2006 14:12:45 GMT Steffen Kaiser <>

On Wed, 24 May 2006, Yutaka Sato wrote:


during my tests, I found three problems, please see below and the 
attached log/conf files.

> In message <_A3236@delegate-en.ML_> on 05/23/06(22:09:12)
> you Steffen Kaiser <> wrote:
> |in short it works:
> What happens without -D_LARGEFILE64_SOURCE ?


gcc -DSTAT64 -x c++ -DQS -DNONC99 -I../include -DSTAT64 -c In function `int set_utimes(const char *, int, int)': aggregate `struct FileStat st' has incomplete type and 
cannot be initialized type `stat64' is not yet defined
make[4]: *** [setutimes.o] Error 1
make[4]: Leaving directory `/mnt/install/linux/delegate9.2.2-pre2a/rary'
make[3]: *** [libx] Error 2
make[3]: Leaving directory `/mnt/install/linux/delegate9.2.2-pre2a/rary'
mkmake: ERROR LOG is left at 
mkmake: ERROR LOG is left at 
make[2]: *** [../lib/library.a] Error 2
make[2]: Leaving directory `/mnt/install/linux/delegate9.2.2-pre2a/src'
make[1]: *** [start0] Error 2
make[1]: Leaving directory `/mnt/install/linux/delegate9.2.2-pre2a/src'
mkmake: ERROR LOG is left at 
mkmake: ERROR LOG is left at 
make: *** [all] Error 2

> escape the problem with the enclosed patch.

This worked - sort of.

Problem 1) Downloaded file is longer than original one.

ncftp /pub/debian-iso/sarge > dir
-rw-r--r--  1 root   4686706688   Jun 13  2005 
-rw-r--r--  1 root   4428851200   Jun 13  2005 
-rw-r--r--  1 root    113541120   Jun 13  2005 
ncftp /pub/debian-iso/sarge > get debian-31r0a-i386-binary-1.iso
debian-31r0a-i386-binary-1.iso:    ETA:   0:00  785.63/373.59 MB   14.79 

However: The returned size is wrong - however, ncftp proceeds downloading 
the file.

I'm not sure, how ncftp tries to get the file size, this one command is 

05/24 08:58:27.87 [22518] 2+0/14: Unknown request: MLST 

but I'm assume ncftp uses the SIZE command. (Same size is reported by 

BUT the downloaded file differs in size, it is 756 bytes longer than the 
original one.

So I tested it further,

dd if=/dev/zero of=test bs=1024 count=175
perl  ~/ >test2 && ls -al test*
-rw-r--r--    1 root     root       179200 May 24 15:14 test
-rw-r--r--    1 root     root       179201 May 24 15:14 test2

cmp -l test*
  65537   0 277
cmp: EOF on test

dd if=/dev/zero of=test bs=1024 count=176
perl  ~/ >test2 && ls -al test*
-rw-r--r--    1 root     root       180224 May 24 15:15 test
-rw-r--r--    1 root     root       180226 May 24 15:15 test2

((test2 is one byte longer!))

cmp -l test*
  65537   0 277
cmp: EOF on test

When I use /dev/random except /dev/zero, the data is shifted by one byte 
from 65537, the 0xFF is inserted.

With count=240, the downloaded file is three bytes longer.
It sort of looks like that there is a problem each 64KB.

I used attached script and explicitly set TYPE I and dumped 
the stuff through STDOUT, so there is no filesystem related stuff. Also, 
the test file is filled with \0's, hence, no CR/LF transformation should 
be done.

Problem 2) The local file is cached.

OK, the host I tried is a Woody box, which is fairly old and I tried to 
check if the downloading stuff is interferred by the old software and used 
another box for download, which is Debian Sarge (current stable). The 
utilities (like cmp) on Sarge do accept 4GB files, those of Woody don't.

The situation is this:

Workstation -> Delegate FTP relay -> Delegate MASTER 
ux-2s04:11011 -> Delegate FTP ux-2s04:2121 -> file:///mnt/mirror/pub

I have a chain of two Delegate-Servers, a generalist (ux-2s04) that DOES 
CACHING and is to tunnel all requests into the world. On the same host, 
the Debian DVD image are located and are served via Delegate FTP.
On an relay some application level proxies forward the requests to the 
generalist. None of them performs caching.

I configured the FTP relay with "MOUNT=/pub/* ftp://ux-2s04:2121/pub/*", 
but through the MASTER=ux-2s04:11011. I guess I have to re-think this 
Master stuff, in order to avoid this problem.

Anyway: The file is downloaded, and the Master caches the file, and when 
it reaches the 2GB boundary, the server drops the connection.

Problem 3) cd works exactly one time

OK, I do this:

ftp relay:2121

cd /pub/debian-iso/sarge
get file
--> OK

BUT when I do:

ftp relay:2121
cd /pub
cd debian-iso/sarge

I get: Could not chdir to sarge/: server said: 
/pub/debian-iso/pub/debian-iso/sarge: No such directory.

This is also true, when I terminate the connection and re-connect again 
too soon, from the logfile I guess that the connection is still hold, once 
dropped, the first CD is successful.

In the master's 2121 logfile this is printed:

05/24 14:01:57.59 [22928] 1+0/8: local echo for PWD: 
05/24 14:01:57.59 [22928] 1+0/8: I-SAY: 257 "/pub/debian-iso" is current 
05/24 14:01:57.59 [22928] 1+0/10: *** 
/pub/debian-iso/pub/debian-iso/sarge/ => 
b/debian-iso/sarge/ ***
05/24 14:01:57.59 [22928] 1+0/10: MOUNTED-TO-LOCAL#UNKNOWN: 
05/24 14:01:57.59 [22928] 1+0/11: *** /pub/debian-iso/ => 
file://localhost/mnt/mirror/pub/debian-iso/ ***
05/24 14:01:57.59 [22928] 1+0/11: MOUNTED-TO-LOCAL: 
05/24 14:01:57.59 [22928] 1+0/12: local echo for PWD: 
05/24 14:01:57.59 [22928] 1+0/12: I-SAY: 257 "/pub/debian-iso" is current 
05/24 14:01:57.59 [22928] 1+0/14: *** 
/pub/debian-iso/pub/debian-iso/sarge/ => 
b/debian-iso/sarge/ ***

The CD seems to append the absolute path to the CWD.



Steffen Kaiser

  admin search upper oldest olders older1 this newer1 newers latest
[Top/Up] [oldest] - [Older+chunk] - [Newer+chunk] - [newest + Check]