freebsd-net · shivansh · Sep 27, 2016 · Oct 31, 2016 · Nov 1, 2016 · Nov 4, 2016
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,7 @@
-# The tests are incomplete
-havercvdfin/
+# Ignore the log files
+error.log
+temp.log
+*tests_list
+
+# Ignore patch files
+patches
diff --git a/ICMP/README.md b/ICMP/README.md
diff --git a/Documents/Report-1.pdf → Progress-Reports/Report-1.pdf b/Documents/Report-1.pdf → Progress-Reports/Report-1.pdf
diff --git a/Documents/Report-2.pdf → Progress-Reports/Report-2.pdf b/Documents/Report-2.pdf → Progress-Reports/Report-2.pdf
diff --git a/Documents/Report-3.pdf → Progress-Reports/Report-3.pdf b/Documents/Report-3.pdf → Progress-Reports/Report-3.pdf
diff --git a/README.md b/README.md
@@ -7,12 +7,34 @@ Regression testing is one of the most critical elements of the test artifacts an
 ## Project description
 
 _packetdrill_ currently supports testing multiple scenarios for TCP/IP protocol suite within Linux. This project aims to design and implement a wire level regression test suite for FreeBSD using packetdrill. The test suite will exercise various states in the TCP/IP protocol suite, with both **IPv4** and **IPv6** support. Besides Linux, the _packetdrill_ tool works on {**Free**, **Net**, **Open**} **BSD**.
-The existing Linux test suite implemented within _packetdrill_ will provide a basis for understanding, and implementation of the FreeBSD test suite. For the current scope of the project, only a subset of the existing test scenarios will be implemented.
+The existing Linux test suite implemented within _packetdrill_ will provide a basis for understanding, and implementation of the FreeBSD test suite. ~~For the current scope of the project, only a subset of the existing test scenarios will be implemented.~~ **Update:** All the scenarios have been implemented along with additional ones.
 
 ## Why Packetdrill?
 
 While valuable for measuring overall performance, TCP regression testing with _netperf_, application load tests, or production workloads can fail to reveal significant functional bugs in congestion control, loss recovery, flow control, security, DoS hardening and protocol state machines. Such approaches suffer from noise due to variations in site/network conditions or content, and a lack of precision and isolation, thus bugs in these areas can go unnoticed. Since _netperf_ is supposed to be more for benchmarking purposes and what we are trying to do is measure correctness, _packetdrill_, which was built with the same mindset, seemed an apt choice for this project.
 
+## Installation
+
+Clone the repository using the following command -
+```
+git clone [email protected]:shivrai/TCP-IP-Regression-TestSuite.git
+```
+
+Now proceed with the [steps for configuring packetdrill](https://github.com/google/packetdrill/blob/master/gtests/net/packetdrill/README).
+
+### Port installation
+The testsuite is available as a [freebsd port](https://www.freshports.org/net/tcptestsuite/) along with [tuexen/tcp-testsuite](https://github.com/tuexen/tcp-testsuite).
+* If you already have packetdrill installed and configured - <br>
+ ```
+ pkg install tcptestsuite
+ ```
+
+* For installing the entire suite with packetdrill - <br>
+ ```
+ pkg install packetdrill
+ pkg install tcptestsuite
+ ```
+
 ## Test Plan
 
 _packetdrill_ supports two modes of testing - local and remote.  A **TUN** virtual network device is used in the local testing and a physical **NIC** is used for the remote testing.
@@ -38,7 +60,6 @@ Local mode is the default mode, and hence the user need not specify any special
 ```
 Executing the above command will give the information about the inbound injected and outbound sniffed packets which can be studied and checked whether in accordance with the expected behaviour. The TUN virtual network device will be used as a source and sink for packets in this case.
 
-
 ### Remote mode testing
 
 On the system under test (i.e the “client” machine), a command line option to enable remote mode (acting as a client) and a second option to specify the IP address of the remote server machine to which the client packetdrill instance will connect must be specified.
@@ -67,46 +88,37 @@ For testing using AF_INET6 sockets with IPv6 traffic -
 ```
 Since the IPv6 headers are 20 bytes larger than the IPv4 headers, the MTU has to be set to 1520 to address the extra 20 bytes, rather than the standard size of 1500 bytes.
 
+### Using the script to automate
+The script [run-tests](https://github.com/shivrai/TCP-IP-Regression-TestSuite/blob/master/run-tests) can be used to automate the tests. The value of the `$packetdrill` variable should be set to the location of the **packetdrill** binary on your machine before running the script. The default path is `/usr/local/bin/packetdrill` in FreeBSD and `/usr/bin/packetdrill` in Linux.
+The following command should be used for executing all the tests -
+```
+sudo sh run-tests <directory/file>
+```
+You can specify the directory for which you want to run the tests. **Note:** If no directory/file is specified, then `pwd` is taken into consideration for generating the list of tests (including all the subdirectories). <br>
+**Note:** `sudo` is required only for running the packetdrill binary. <br><br>
+**Update:** Logging functionality is now added in the automation script. After a single run, all the errors are placed in `error.log` in a clean format for easy debugging. <br>
+
 ## Scenarios covered
 
-|**Scenario**|**Number of tests**|**Result**|
-------------|:-------------------:|:----------:|
-|ICMP|1|Passed|
-|Blocking system calls|2|Passed|
-|Fast Retransmit|1|Passed|
-|Early Retransmit|1|[Failed](https://github.com/shivrai/TCP-IP-Regression-TestSuite/tree/master/early_retransmit#test-for-early-retranstmit)|
-|Fast Recovery|1|Passed|
-|init_rto|1|Passed|
-|Initial window|1|Passed|
-|PMTU discovery|1|Passed|
-|Retransmission Timeout|2|Passed|
-|Socket Shutdown|3|Passed|
-|Undo|2|Passed|
-|Connect|1|Passed|
-|TCP options establishment|5|Passed|
-|AIMD|1|Passed|
-|TIME-WAIT configuration|1|Passed|
-|Selective Acknowledgements|1|Passed|
-|Connection Close|5|Passed|
-|Simultaneous Close|1|Passed|
-|RESET from synchronized and <br> non-synchronized states|7|Passed|
-|MSS|8|6/8 Passed|
-|Receiver RTT|2|Passed|
-|TCP timestamps|*|Passed|
+|**Scenario**|**Number of tests**|**Result (FreeBSD)**|
+-------------|:-----------------:|--------------------|
+|[ICMP](icmp)                    |1 |Passed
+|[Socket API](socket-api)        |17|Failed: [1](socket-api/close/README.md)
+|[TCP Mechanisms](tcp-mechanisms)|12|Failed: [1](tcp-mechanisms/early_retransmit/README.md)
 
 ## Possible Scenarios
-* ~~Test for half-open connection (last ack getting lost).~~ **Update -** [close-last-ack-lost](./close/close-last-ack-lost.pkt)
+* ~~Test for half-open connection (last ack getting lost).~~ **Update -** [close-last-ack-lost](socket-api/close/close-last-ack-lost.pkt)
 
 ## Future Plans and Work
-There is a huge scope for work yet to be done in this project, and I am not stopping anywhere in the near future. The final goal is to make this test suite exhaustive so that it can be easy for FreeBSD developers for checking the authenticity of the network stack, and that any misbehavior can be easily found out and rectified. The number of scenarios that can be added are innumerable, and the existing implemented set will be kept expanding and perfected. <br>Some of the tasks which can be listed as of now are -
- * Once we are successful in adding support in **tcp_info()** for checking window size, scenarios such as sliding window protocol, zero window handling and zero window probing can be successfully tested.
+There is a huge scope for work yet to be done in this project, and I am not stopping anywhere in the near future. The final goal is to make this test suite exhaustive so that it can be easy for FreeBSD developers for checking the authenticity of the network stack in a rigorous manner, and that occurrence of any misbehavior can be found out and rectified easily. The number of scenarios that can be added are innumerable, and the existing implemented set will be kept expanding and perfected. <br>Some of the tasks which can be listed as of now are -
+ * Once we are successful in adding support in **tcp_info()** for checking window size, scenarios such as sliding window protocol, ~~zero window handling and zero window probing~~ **Update:** [3d359c](https://github.com/shivrai/TCP-IP-Regression-TestSuite/commit/3d359cb60a2c682dfae28d640eedfe9d7e2ef361) can be successfully tested.
  * Adding support for urgent pointer in _packetdrill_.
  * _packetdrill_ currently supports testing only a single connection at a time. An attempt will be made to patch it to support multiple concurrent connections.
  * The current remote mode available in _packetdrill_ allows testing a remote host provided there is already an instance of _packetdrill_ running on it. There is not yet support for testing a remote host that does not have _packetdrill_ running. One such approach for enabling support for this can be that instead of getting command line arguments and the script over a TCP connection, the current instance can get it directly. Hence, the logic for handshake with the client will be removed, the packets will be injected and the client will wait for inbound packets.
 
-**Keep in touch with the latest updates in the project via the [FreeBSD-wiki](https://wiki.freebsd.org/SummerOfCode2016/TCP-IP-RegressionTestSuite/).**
+**Keep in touch with the latest updates in the project here or via the [FreeBSD-wiki](https://wiki.freebsd.org/SummerOfCode2016/TCP-IP-RegressionTestSuite/).**
 
 ## Acknowledgements
-I cannot thank my mentor **Hiren Panchasara** enough for all the help and support which he has given in the entire duration of the project. Working in his guidance was full of fun and challenging. I would be grateful enough to receive his guidance in the future too while I continue to work on this project. <br>
+I cannot thank my mentor **Hiren Panchasara** enough for all the help and support which he has given in the entire duration of the project. Working in his guidance was full of fun and challenging. I would be extremely grateful to receive his guidance in the future too while I continue to work on this project. <br>
 I also thank **Michael Tuexen** for all his help and guidance during the project. <br>
 I would also like to thank the FreeBSD community for accepting this project, which gave me a chance to gain a lot of knowledge about the community and open source in general.
diff --git a/close/README.md b/close/README.md
diff --git a/icmp/README.md b/icmp/README.md
@@ -0,0 +1,64 @@
+# Test for handling incoming ICMP messages
+
+## Status
+
+|  Num  |	Abbrev.	  |	Description               | Status
+|:-----:|:---------:|:--------------------------|:-----------------:
+0	|echorep	 	|Echo reply                 |:white_check_mark:
+3	|unreach		|Destination unreachable    |
+4	|squench		|Packet loss, slow down     |
+5	|redir		  |Shorter	route exists      |
+6	|althost		|Alternate host address     |
+8	|echoreq		|Echo request               |:white_check_mark:
+9	|routeradv	|Router advertisement       |
+10	|routersol	|Router solicitation        |
+11	|timex		  |Time exceeded              |
+12	|paramprob	|Invalid	IP header         |
+13	|timereq		|Timestamp request          |
+14	|timerep		|Timestamp reply            |
+15	|inforeq		|Information request        |:white_check_mark:
+16	|inforep		|Information reply          |:white_check_mark:
+17	|maskreq		|Address	mask request      |:white_check_mark:
+18	|maskrep		|Address	mask reply        |:white_check_mark:
+30	|trace		  |Traceroute                 |
+31	|dataconv	  |Data conversion	problem   |
+32	|mobredir	  |Mobile host redirection    |
+33	|ipv6-where |IPv6 where-are-you         |
+34	|ipv6-here	|IPv6 i-am-here             |
+35	|mobregreq	|Mobile registration request|
+36	|mobregrep	|Mobile registration reply  |
+39	|skip		    |SKIP                       |
+40	|photuris	  |Photuris                   |
+
+|  Num | Abbrev.        |   Type        | Description                         | Status
+|:----:|:------------:  |:-------------:|:-----------------------------------:|:------:
+0	|net-unr		    |unreach	      |Network unreachable                  |:white_check_mark:
+1	|host-unr	      |unreach	      |Host unreachable                     |:white_check_mark:
+2	|proto-unr	    |unreach	      |Protocol unreachable                 |:white_check_mark:
+3	|port-unr	      |unreach	      |Port unreachable                     |:white_check_mark:
+4	|needfrag	      |unreach	      |Fragmentation needed but DF bit set  |:white_check_mark:
+5	|srcfail		    |unreach	      |Source routing failed                |:white_check_mark:
+6	|net-unk		    |unreach	      |Network unknown                      |:white_check_mark:
+7	|host-unk	      |unreach	      |Host unknown                         |:white_check_mark:
+8	|isolate		    |unreach	      |Host isolated                        |
+9	|net-prohib     |unreach	      |Network administratively prohibited  |:white_check_mark:
+10	|host-prohib	  |unreach	      |Host administratively prohibited     |:white_check_mark:
+11	|net-tos		    |unreach	      |Invalid TOS	for network             |:white_check_mark:
+12	|host-tos	      |unreach	      |Invalid TOS	for host                |:white_check_mark:
+13	|filter-prohib	|unreach	      |Prohibited access                    |
+14	|host-preced	  |unreach	      |Precedence violation                 |:white_check_mark:
+15	|cutoff-preced	|unreach	      |Precedence cutoff                    |:white_check_mark:
+0	|redir-net	    |redir	        |Shorter route for network            |
+1	|redir-host	    |redir	        |Shorter route for host               |
+2	|redir-tos-net	|redir	        |Shorter route for TOS and network    |
+3	|redir-tos-host |redir	        |Shorter route for TOS and host       |
+0	|normal-adv	    |routeradv      |Normal advertisement                 |
+16	|common-adv	    |routeradv      |Selective advertisement              |
+0	|transit		    |timex	        |Time exceeded in transit             |
+1	|reassemb	      |timex	        |Time exceeded in reassembly          |
+0	|badhead		    |paramprob      |Invalid option pointer               |
+1	|optmiss		    |paramprob      |Missing option                       |
+2	|badlen		      |paramprob      |Invalid length                       |
+1	|unknown-ind	  |photuris       |Unknown security index               |
+2	|auth-fail	    |photuris       |Authentication failed                |
+3	|decrypt-fail	  |photuris       |Decryption failed                    |
diff --git a/ICMP/icmp-all-types.pkt → icmp/icmp-all-types.pkt b/ICMP/icmp-all-types.pkt → icmp/icmp-all-types.pkt