failure to create a directory in dCache gridftp server

[DmitryLitvintsev]

Hi,

we observe this error:

[litvinse@uqbar trunk]$ uberftp stkendca17a -P 2811
220 GSI FTP door ready
200 User :globus-mapping: logged in
UberFTP> mkdir new_dir
new_dir: 550 File not found
UberFTP>

we send single line "550" reply on MLST command which follows RFC 959. Unerftp client
seems to expect multi-line reply "550- .... \n 550 ..."

Please fix.

[JasonAlt]

Per RFC 3659 (http://datatracker.ietf.org/doc/rfc3659/?include_text=1)

7.2. Format of MLSx Response

The format of a response to an MLSx command is as follows:

  mlst-response    = control-response / error-response
  mlsd-response    = ( initial-response final-response ) /
                     error-response

  control-response = "250-" [ response-message ] CRLF
                     1*( SP entry CRLF )
                     "250" [ SP response-message ] CRLF

  initial-response = "150" [ SP response-message ] CRLF
  final-response   = "226" SP response-message CRLF

  response-message = *TCHAR

  data-response    = *( entry CRLF )

  entry            = [ facts ] SP pathname
  facts            = 1*( fact ";" )
  fact             = factname "=" value
  factname         = "Size" / "Modify" / "Create" /
                     "Type" / "Unique" / "Perm" /
                     "Lang" / "Media-Type" / "CharSet" /
                     os-depend-fact / local-fact
  os-depend-fact   = <IANA assigned OS name> "." token

I'm a bit jet lagged so correct me if I'm missing something but it appears that MLST responses are multi line.

[DmitryLitvintsev]

Hi Jason,

Yes, 250 reply is multi lined, but there is nothing about error response in the above.

For a successful MLST we indeed reply "250- ...
....
250 ..."

from 7.2.1. Error Responses to MLSx commands of the quoted document:

...Giving a name that does not
exist, or for which access permission (to obtain directory
information as requested) is not granted will elicit a 550 reply. ...

550 reply is not required to be multilined.

A circumstantial evidence:

A regular ftp client executed against our so called weakftp door is successful.
(weakftp (aka password authenticated ftp) door uses the same implementation of the MLST as our gridftp door):

ftp fnisd1 22126
Connected to fnisd1.fnal.gov.
220 Weak FTP door ready
Name (fnisd1:dmitrylitvintsev): litvinse
331 Password required for litvinse.
Password:
230 User litvinse logged in
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> mkdir test_directory
257 "/test_directory" directory created
ftp>

[paulmillar]

Although RFC 959 allows multi-line responses, RFC 3659 has a stronger requirement. 3659 has BNF defining 'error-response' as

  error-response = error-code SP *TCHAR CRLF
  error-code     = ("4" / "5") 2DIGIT

Therefore 'error-response' MUST NOT (RFC 2119) be a multi-line response and the Globus FTP server doesn't conform to RFC 3659.

You may like to open a bug against Globus to ask them to fix their FTP server, but in the meantime I suggest you support both multi-line and single-line error responses.

[JasonAlt]

Thanks for the clarification although it is still not clear exactly what problem you are having. The latest version of UberFTP does not have any explicit requirements for the number of lines in the MLST error response; it assumes any code >= 400 is an error and appends the server's response to the requested path name. And even if UberFTP did have this requirement, how does that relate to this bug's description "failure to create a directory in dCache gridftp server"?

[DmitryLitvintsev]

The issue seems to be with uberft expecting "XXX- .... XXX" reply. Rather that "XXX ...." reply.
The hyphen at the end means multi-line response follows.

I checked that if instead of "550 File does not exist" I reply "550- File Does not exist\n500 File Does not exist" then uberftp client works fine (independently from of the reply is actually being multi-lined (that is if I skip "\n" from above string)). I downloaded and built uberftp client from here.

[JasonAlt]

I do not see any code paths that suggest that multi line vs single line error responses for MLST will affect mkdir. However, I do see that if a server's error response to MLST contains "No such file or directory", the MLST will be treated as successful (just that the target does not exist). This was done to attempt to distinguish fatal errors (EPERM) from expected errors (ENOENT) so that we could determine how and when to proceed with an operation. With dCache, the server error appears to be 'File Does not exist' and this is enough to cause MLST to appear to have failed and so MKD is never performed.

But this is not what you are reporting which is what is confusing. Can you run UberFTP in debug mode so we can see what commands are being issued by the client? Or can you give me a dCache account to debug this directly?

[DmitryLitvintsev]

OK, I will provide it in a few minutes.
The reason I started to talk about multi-line response is because I looked in debug mode
as to what vanilla globus gridftp server return on mkdir command:
....
... ts=2012-10-15T19:46:58.554388Z id=17599 event=globus-gridftp-server.session.message sender=server msg="500-Command failed : System error in stat: No such file or directory 500-A system call failed: No such file or directory 500 End. "
....
So I thought, aha, uberftp is OK with parsing "XXX- .. XXX" response, but fails on "XXX ... " response.

[DmitryLitvintsev]

here is debug output:

UberFTP> mkdir uberftp_test2
MLST .
Encoded cmd: ENC FwMAACBZqiq2bq/yuotclXVTFMWpOBoU5kU1E4arshKc6UBdkg==
Encoded resp: 633 FwMAAMDSE2xfbO9bvnl6I64/HKy9kiFU0BkZammQ+KGBHBlxFLRt3ukrtC8ePXZs570nIWg/JXA1ErOZA7qPw7izSOS894Vh14MgNwuqDibbsjZhruD52ShfQa8mIfSQfkA1mu63o5nyzj2UvZ0kz0xkU2dMEiVjkI17Xq488M4NOE6+CwZMJd9JlEnGtX1j1EEpokf3EYitZbe+uixDNNJ2z1Vo94909rukjpTVMGXXuxd7o04GrYxiuG9hC880DPldz2g=
250- Listing .
UNIX.group=3200;Perm=celm;Type=dir;UNIX.mode=755;UNIX.owner=8637;Modify=20121017114537;Unique=0000429A808057A249989337D15223852E24; /litvinse
250 End
MLST uberftp_test2
Encoded cmd: ENC FwMAADDHkV++7is8T0hreceW3+R0F/Pe28367yMS49DfKZDjXBAGA9cfrclITUbAsUdwVS8=
Encoded resp: 633 FwMAADDSMBBhzA+XUjt+kXDQ3tzcmN0pLI6qspkDwXCTEpodCFY+AYML6aKjy3Kzohccbpk=
550 File not found
uberftp_test2: 550 File not found

[DmitryLitvintsev]

for completeness, this is the same attempt with vanilla globus gridftp server:
UberFTP> mkdir uberftp_test2
MLST .
Encoded cmd: ENC FwMAACCtx1e46flrEFNcFw3rDbrOgDdUUq/xE29pMHZw/KDvOw==
Encoded resp: 632-FwMAADDBb5rGRdAYmG/xoLACfmiak7a5FPqJX/ckMYSZRKfuIfuyXbKKyu6gBmMEbjf4+Y8=
632-FwMAALD5lt7tyDxTtgqEGolTJ9BSjCL+e+5W3fYesQtAAi9GDN8MAkzpatCGUIPsUvVmo2aONDeSrBjrCJ0c8UYwl0FrKDtF+Fz0Kh7AbcJMqNJwXmaYWTCW0bX8P8Nf7o3wqY5LaQkcQFCtTvHq7GQgF1vVLXehleBjAQd9v+EhyYS1UJPug+K+rfnXTxtNCw+aTHOwJra6+tdHEMqjkVgpCV/KZP81/HHzV/2gRAPLN/Darg==
632 FwMAACC22aSCNjAAuNip9A+i9q8CbMUGtQ9Q+n7sbUtzhZhwMQ==
250-status of .
Type=dir;Modify=20121022195046;Size=4096;Perm=el;UNIX.mode=0550;UNIX.owner=root;UNIX.uid=0;UNIX.group=root;UNIX.gid=0;Unique=805-374001; .
250 End.
MLST uberftp_test2
Encoded cmd: ENC FwMAADDiti7iYvb1Bh0ydb+NOdAYUPR88sjZWh170d8zIly+BQSmjdpPfh/+5lbjZwq6Q50=
Encoded resp: 632-FwMAAGBycX92D6viyXRa3f1MLlUTjQGxlSBdBuI1Htg+cLlqa5vfOapRjZWDKL8aExSU1rLnTW7uHy5fK6GoVyXKVB0k9yjKfX/qtI1SSJ06GLblD9k/IzY046gxi4Q9ZUtv4sE=
632-FwMAAFBfRiwlwwCQNW//e94vrTuUBIPLEX5DvAi499iLIf/Pf3UuuhSaJCcqGK86l19wiJRlbkeVhhRqIhMKb3QMTTxUO63tes75mAwwxdP0Y4brUA==
632 FwMAACBn+jEjTKDhtJFv/qlOZHZu18wRVMT0YfQ9MKOYzupwzQ==
500-Command failed : System error in stat: No such file or directory
500-A system call failed: No such file or directory
500 End.
MKD uberftp_test2
Encoded cmd: ENC FwMAADCz/7Y+QOVWV3v3w4G5UlbTzUgHClNgVjPlp70nA9yVyMy2lfhKH1w+uxfuaS8ngik=
Encoded resp: 632 FwMAAFAv1rLyKc3xWqUFTBEl4RKxipcjWJg7Cpaxcc71isRwStoH81OFJc1CsUX5PyLNGDuDrPGS36fAWq8xhygFY991G1Rn50RT64mINWLindLxnw==
257 Directory "/root/uberftp_test2" created successfully.

I see multiline "500-" response. (why is it 500 is unclear to me. It should be 550 according to the RFC)

[DmitryLitvintsev]

Ah, I get it. Having replaced "File not found" w/ "No such file or directory" I can successfully create directory using uberftp.

[JasonAlt]

OK terrific. Seems that we can either patch dCache or patch UberFTP or patch both. Patching the server will make it work with all versions of UberFTP that are floating around (but possibly break something else). Fixing UberFTP will only work for users with the (next) latest version.

I'll update github with the source changes first thing in the morning.

[paulmillar]

Now I'm confused.

The client does an MLST on a directory that it expects not to exist (the user has requested creating the directory; this will only work if the directory doesn't already exist). When this command fails, the client parses the string-part of the return code (which is intended for human consumption) looking for a particular string and only proceeds if that string is found.

This doesn't seem a very robust approach: AFAIK, the strings returned by FTP servers under error conditions aren't defined anywhere.

I guess the first question is why is the client issuing a MLST on the directory in the first place? Why not simply issue the MKD command?

Also, I also don't understand what you mean by "fatal errors (EPERM)" and "expected errors (ENOENT)". These terms are not defined in RFC 959 or 3659.

[JasonAlt]

There is very little about the FTP protocol that I would consider robust. And that is doubly true for anything concerning error responses. Most everything that is relatively complicated in the development of a FTP client revolves around creating a sane environment for users through an otherwise clumsy protocol.

The reason things work the way they do in the client is to support batch mode and regular expressions (and as a general convenience for users). If I were to write a shell script that had to support regexps and directory creation in a batch environment (meaning things should just work if possible, stop only when absolutely necessary) then I would (1) expand the regexp, (2) determine if any directories already exist, (3) create any directories that do not exist and (4) only report and break on any real error.

The exact same is true for FTP. I expand all regexps (mkdir ~/*/data), figure out what exists and what doesn't and create what needs to be created. Unfortunately, FTP doesn't allow me to distinguish EPERM from ENOENT without parsing the human readable portion of the error response. Distinguishing between error types is particularly useful when batched operations have to be restarted due to some previous failure like a dropped connection. The script should be allowed to be re run and complete without failure as opposed to failing because a previously created directory now exists.

This brings us to the problem found with UberFTP & dCache. The client can not distinguish the error type and so it expects the worst and exits. The MKD is never performed.

Hope that helps.

[JasonAlt]

The fix has been pushed to the develop branch. Please verify it then I'll tag it as 2.7 beta 5. I'm holding off on the official 2.7 release until the corresponding DSI work is done.

[paulmillar]

Hi Jason,

There's several interesting points in what you say; however, I'll try
to limit it to (what I believe is) the crux of the issue.

"The script should be allowed to be re run and complete without
failure as opposed to failing because a previously created directory
now exists."

Note that this requires that operations (like creating a directory)
are idempotent. This isn't true for POSIX shell 'mkdir' command and
(from my reading) goes against RFC 959 (see Appendix II).

However, if you wish UberFTP to treat mkdir as idempotent, then of
course that's perfectly fine. However, one should recognise that this
isn't the semantics of FTP and that the client needs to do extra work
to achieve this.

I'm guessing the semantics of 'mkdir' in UberFTP is something like
"ensure that the named directory(-ies) exist, creating them if
needed". The command should succeed if the directory/-ies exist and
fail if (and only if) the client is unable to create a missing
directory. This would be idempotent (modulo transient errors) and, to
me at least, kinda makes sense.

Is this the correct semantics of mkdir in UberFTP?

If so, one can achieve this without distinguishing between permission
denied and directory-doesn't-exist errors for MLST operations.

1. Expand any glob the user supplies to build a list of directories
   that are required to exist. (From what you've said, I believe
   UberFTP does this currently).
2. Issue the MKD command on each directory from step 1. Remembering
   which ones failed.
3. For each failed directory, use MLST to check the directory exists
   and it is a directory.

If any directory fails both the MKD and MLST operation then fail the
mkdir operation in UberFTP.

Of course, we can change the message we return for MLST to match what
you expect (we already have a patch for this), but what happens if a
different client expects a different string?

For me, this is a client-bug: we can work around it in dCache, but it
should be fixed in UberFTP.

HTH,

Paul

PS. Looking at the dCache code, we currently return 200 on successful
MKD, which is wrong: it should be 257, with the new directory's path
in quotes. Dmitry has a fix for this. Also, it looks like we return
553 if the directory already exists; this, too is wrong (should be
521). sigh