Factor out the WiFiClient/Server objects

[devyte]

The WiFiClient/Server should be factored out. This can be done by allowing a constructor that receives the object already prepared beforehand. An example is the http client. Alternatively, you can use an approach like the webserver.
The CAs or fingerprint or whatever can then be set up before passing in.
As in other such cases, the FTP client/server would not own the obkect passed in and so shouldn't destroy it.

[dplasa]

I need to think about that.
At first, it sounds reasonable but then questions arise. FTP uses a control connection and for every transfer a separate data connection that needs to be established every time, data (e.g. a file or a directory listing) is transferred.

• in active mode, the FTP Server opens a connection to the client with parameters set by the client
• in passive mode, the FTP Server needs to listen on a port, that the server tells the client to connect to
• if the control connection is encrypted (also the data conenction needs to be is encrypted, or at least that's what vsftpd requires)

Not sure, what a clean approach here would be.

[devyte]

From what I can tell, the following aspects should be considered:

• for the WiFiServer, you have to allow setting up the secure settings externally. In passive mode, the 2nd server would likely share the same settings.
• for the WiFiClient used by the server in active mode, thought must be given to how that would be constructed or set up.
• for the WiFiClient used by the client, you have to allow setting up the secure settings externally.

There are examples in the core for using and setting up the secure client/server, including a certificate store.

This issue is long term, and will likely require quite a bit of experimentation :p

[dplasa]

I'm playing around with some thoughts... curious to know your 2ct ;-)

General:

*Should the limitation for 1 connection at a time be removed? This is what stops curlftpfs from working... Or would that be too much effort for an old protocol?

Secure connections:

• The FTP Server could own the control WiFiSecureServer, provide an access function getServer() (like the webserver), after receiving a PASV command from a connected client, it copies / shares (?) the secure settings to it's second (data) server and starts listening.
• If we keep the 'one at a time' status, we could also just re-use the control server after a control connection has been made to listen as the data server on another port. After the current control connection is closed, it switches back to act as control server.
• In active mode, whenever the server has to open a (secure) connection to the ip/port set by the client, the server could call a provided callback function so the WiFiSecureClient could be configured before actually making that data connection.

[devyte]

@earlephilhower is the right person to ask about the secure client/server, and whether the set up propagates/is shared when copying a server object.
Maybe there should be an explicit method for that.
In general terms, I can say that the 16KB buffer is NOT shared, and that the ~6KB stack IS shared, but the person to discuss all of this with is @earlephilhower.

Because of the 16KB buffer, it's likely that 1 connection at a time is all that is possible, at least with secure comms and internal sram.
I suggest looking into MFLN. If the user controls both ends, e. g. both are an esp, or the non-esp side supports it, making use of MFLN would reduce ram use significantly.

[earlephilhower]

FTP's inane setup with different dynamic control and data connections may be an issue when using WiFiClientSecure.

WiFiServerSecure doesn't use much mem, but when you accept a connection it generates a new WiFiClientSecure object you use to talk to the other side, assuming the encryption handshake went well. That object you can copy around the same as a WiFiClient. Probably ~20kb worth of per-object data w/o MFLN. If MFLN is done (requires the client to request it, your ESP8266 server cannot due to the SSL protocol) that can go down to ~5k-ish.

There is no concept of what you're trying to in any SSL:

...after receiving a PASV command from a connected client, it copies / shares (?) the secure settings to it's second (data) server and starts listening...

SSL handshakes are required on any TLS connection. You can't copy things to short-circuit them. If you need 2 connections on different ports, you need 2 handshakes and 2 separate state objects. So w/o MFLN it's very unlikely you will be able to do 2 parallel connections.

There is always a single 6K stack allocated whenever a WiFiServerSecure or WiFiClientSecure object exists, as @devyte said.

[devyte]

@earlephilhower the question isn't whether the state can be copied over, but whether the setup can, i. e. fingerprint, CA, whatever. Or have a method so that one object can be set up based on another that is already set up. Is that possible? Does it make sense?

[earlephilhower]

No, there's no way to copy over the settings. You'll need to make the new object at the new data port and apply the same settings calls (cert, mfln, etc.).

[d-a-v]

Apparently only the stream mode is used by this implementation which makes it difficult to have a control on the packet size, which in turn prevents to use MFLN.

There is however a block mode which seems, when implemented, to limit the size of the transfered data blocks. It would suit MFLN and allow to handle two ssl connections in ram (6KB + 2*MFLNlimited).

I cannot tell about usefulness of spending time in trying to implement FTPS. At least not before further understanding of the block mode.
However, I have to say that there is a little number of protocols which are at the same time quite simple and widely used / implemented. FTP is one of them. With tools like filezilla, cyberduck, (I love curlftpfs - it allows to see the esp as a network drive -, and mc, but they are less widely used), it is very easy to transfer file in both direction on the ESP. HTTP allows to download easily, but I hardly think of a widely-used-common-tool that would allow to upload files. I currently use curl in some personal POSTing scripts which is less sexy (and not easy for a new or average user).

[dplasa]

Details can be found in https://tools.ietf.org/html/rfc959. Reading about STRU(cture) {File, Record, Page} and MODE {Stream, Block, Compressed} it feels to me like of a tale from old times when the internet was made from IBM mainframes run from it's own power plant ;) the FTP protocol imho clearly suffers from it's ancient protocol design. It seems straight forward on first hand but a whole bunch of complexity arises from nowadays somewhat obscure transmission modes. Even vsftpd doesn't bother and just supports Stream Mode & File Structure - my guess for the sake of keeping it simple.

With that said, from what I read at first glance, I don't really understand the difference between a Stream mode (in which the server could sends tcp packets as little in size as it wishes) and Block mode (in which the server breaks the stream into smaller chunks and sends them, again via tcp, I assume).

Correct me, if I'm wrong: In my understanding, the size data that the server would write to a tcp connection, doesn't influence the size of the memory involved. If you write big chunks, the tcp stack just breaks it into more packets. Also, in my understanding of @earlephilhower, the amount of memory used for SSL over tcp is set by the SSL connenction handshake and again does not relate to the size of data, that is transferred over that SSL connection. And worse, the only way to make this memory small, is if the client requests it to be small.

[d-a-v]

You are right about TCP. Peers can use whatever buffer size they wish, everything is buffered by the protocol or hold at its source until needed. That's why one should always check read() and write() return values - however in esp8266-arduino write, everything is copied / buffered (unless in sync=slow mode) so size passed to write() should not be too large.

In SSL mode, we can say the same only when MLFN is enabled and negociated.
If it is not enabled on both side of the link, then when one side writes a large chunk and the other read a smaller size: the difference is lost (or an error is thrown I don't know).

[earlephilhower]

In SSL mode, we can say the same only when MLFN is enabled and negociated.
If it is not enabled on both side of the link, then when one side writes a large chunk and the other read a smaller size: the difference is lost (or an error is thrown I don't know).

One minorcorrection. With TLS one side sends a chunk of whatever size it wants, but if you don't receive and store that whole chunk you can't decode anything. It's like a block cipher with variable block sizes.

So, w/MFLN if the other side sends a block larger than you can accept (i.e. the buffer size you set) then you can never decode it and the connection will die at that point. What size you read() out of the decoded chunk is up to you and is immaterial as far as TLS cares. On the current implementation, whatever size you write() is the size of TLS packet that will be generated (unless > than the 512b default xmit buffer, in which case it will be fragmented in the TLS level to multiple TLS packets)