Serving Huge Files Over 4GB by DeleGate
Serving huge files over 4GB via FTP or HTTP was supported in DeleGate/9.2.2. There were two fundamental issues solved in the version 9.2.2 to enable handling huge files:
- Manipulating a file over 2GB (OS dependent)
- Internal representation of the size of data over 4GB
- Swift response to the abortion of data transfer
- Restarting data transfer from a position beyond 4GB
- Logging the transfer, abortion, and restarting of huge data over 4GB
NOTE: Currently, this feature to manipulate files over 2GB is not enabled by default on several operating systems including Linux (maybe non-BSD origin OS) where "off_t" is in 32bits integer. To enable it, add "-DSTAT64" to the "CFLAGS" variable for "make", as follows for example.
make CFLAGS="-DSTAT64 -O"
If you encountered a problem during the make,
be sure that you are using a recent Gcc at least after 3.X.
At least I succeeded it on RedHat9 (gcc 3.2.2) and Debian3.1 (gcc 3.3.5).
On Windows (with VC++), this feature is enabled by default. In other operating systems, including MacOSX, FreeBSD, and OSF/1, this feature is supported inherently.
- Manipulating a file over 2GB
There are operating systems like Linux families in which a file larger than 2GB is not supported by default. It is because system calls to handle a file, including stat(),lseek(), and ftruncate() is based on "struct stat" using the basic type "off_t" to represent a file size is in 32 bits. In those systems, to manipulate a large file over 4GB, there is another set of file manipulation system calls as stat64(),lseek64(),ftruncate64() and so on based on the extended structure "struct stat64", in which a file size is respresented by "off64_t" of 64 bits integer.
- Internal representation of the size of data over 4GB
Most of platforms have 32bits integer, and DeleGate have used "int" to represent the size of data or file. With "int" of 32bits, at most 4GB data can be represented, and if it is interpreted as "signed", it becomes 2GB at most.
A possible solution is using 64bits int overall in DeleGate, but I feel it might be overshooting for DeleGate. Also, there might be codes in DeleGate which is coded expecting that "int" is in 32bits (as in the counter file for example).