brings knowledge and systems to Primus (#1075)

* initial implementation of the new modules

* publishes and documents the new interfaces

* fixes a bug in the has_component function

* translates the binary program linker into a component

deprecates the old add_component in favor of the new Components
module, starts updating the registration functions. Though tests
should work now.

* catches exceptions raised from init

* switches to the new component registration inteface

except the observations, as I will touch them in the other branch.

* restores the old behavior of the add_component function

there is no other way to preserve the user interface.

* implements restricted mode

* rectifies the start/stop observations

* adds name to start/stop events, hides more of transformer's effect

* exposes machine-kill and system-stop observation to Lisp

those signals are much better for the cleanup/teardown tasks
as they are executed in the restricted mode

* mark unvisited nodes with dead

we are getting more interested in terms that we didn't visit rather
than in terms we have visted.

* tests that we have 100% coverage in the taint test

we have three xfails, which we will rectify later.

* the promiscuous mode and greed scheduler now respect Term.visited

So that if a term is already visited we won't fork on it and won't
loose statistics.

* adds the progress and respect with-repetitions in run-in-isolation

Run will now report progress after each new subroutine and, if run
in the isolation mode, will not enter the already visited entry points

* rectifies the kill observation

it was called in a context of a different machine

* fixes the restricted mode

the restriction was checked using the local state, not the local,
therefore only a fork was restricted.

* exposes the run function for the legacy main system

* switches run to run the Machine.Make(Knowledge) monad

the slowdown is only 30%

* adds -O3 to the root OMakefile

apparently it boosts performance both on vanilla and flambda compilers
by about 25-30%

* drops the run function, it is better to expose the system

and use the existing runners.

* introduces Jobs, enables parameterization of systems in Jobs queue

* switches run to the usage of the Job Queue

* adds an option to run only marked subroutines

the idea is that static analysis before run can mark certain entry
points as interesting and they then will be automatically chosen by
the run plugin for further analysis if `marked-subroutines` is used
as the entry point.

* preliminary support for loading and running systems

* drops the legacy_system function uses the repo instead

* adds system/component printing facilities

rectifies names and descriptions

* initial checking of the set of system definitions

* a bit of cleanup

* adds to run the capability to run systems

* a little bit of renaming, plus extended the callbacks on Jobs

so that we can render the progress bar correctly.

* makes observations observable

via the primus-observations command :)

They are also now packaged.

* documents the interpreter observations

* removes the debugging observation-blocked observation

* describes all the observations

* adds a few more docs and deprecates Machine.{init,finished}

as well as rewires existing analyses to more applicable observations.

* adds more docs.

* polishes the primus documentation

* adds the new plugin to opam/opam

* adds systems to the testsuite

Both run and taint tests are now run twice, one for the
bap:legacy-main system and once for the system that corresponds to
that test. For run it is `bap:stubbed-executor` and for taint it is
`bap:reflective-taint-analyzer`.

* changed Result.t to result

* expects all w64 taint-no-deadcode to fail

Probably, we do not have enough symbolic information. But in any case,
we will investigate it later. (The original idea was to add the
--read-symbols-from option, but we don't have symbols for windows
binaries in the testsuite)

* moves taint finalization to the machine-kill observation

All finalization procedures should be invoked in the restricted
mode (i.e., use either machine-kill or system-stop) observations,
as otherwise they risk to have some unpleasant interactions with
other components, e.g., with bap:limit that will kill the machine
when it does any new operation (which increases the machine clock).

In this case it was an infinite loop, as the taint finalization was
attached to the halting observation and was reflected to Primus
Signals which had methods that were doing quite a few
operations, which, in turn, woke up the limited that was using the
halt operation to halt them, and here we go again.

Author: ivg

OMakefile

@@ -33,6 +33,7 @@ include _oasis_lib.om
 OCAMLFLAGS_ANNOT = -annot -bin-annot
 OCAMLFLAGS += $(OCAMLFLAGS_ANNOT)
 OCAMLFLAGS += -opaque
+OCAMLOPTFLAGS += -O3
 
 # Until this point we allow to override variables via the command-line.
 # That means all initializations from above can be changed by omake arguments

lib/bap_primus/bap_primus.ml

@@ -8,19 +8,29 @@ module Std = struct
     module Interpreter = Bap_primus_interpreter
     module Time = Interpreter.Time
     module Linker = Bap_primus_linker
+    module Value = Bap_primus_value
+    module Memory = Bap_primus_memory
+    module Observation = Bap_primus_observation
+    module Lisp = Bap_primus_lisp
+    module Analysis = Bap_primus_analysis.Machine
+    module System = Bap_primus_system
+    module Job = System.Job
+    module Jobs = System.Jobs
+    module Info = Bap_primus_info
+    module Components = System.Components
     module Machine = struct
       module type State = State
       include Bap_primus_machine
       type 'a state = 'a State.t
       include Bap_primus_main
+      let finished = System.fini
+      let init = System.init
     end
-    module Value = Bap_primus_value
-    module Memory = Bap_primus_memory
-    module Observation = Bap_primus_observation
-    module Lisp = Bap_primus_lisp
     type generator = Generator.t
     let sexp_of_value = Value.sexp_of_t
     let value_of_sexp = Value.t_of_sexp
     let compare_value = Value.compare
+    type system = System.t
+    type info = Info.t
   end
 end