Skip to content

Commit 11d5903

Browse files
peterhanilypeterhanily
committed
github-inital
0 parents  commit 11d5903

27 files changed

+2452
-0
lines changed

‎ReadMe.md

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
Fusion as a Service (FaaS)
2+
===================
3+
--------
4+
5+
Install Node.js
6+
----------------
7+
* sudo apt-get install python-software-properties python g++ make
8+
* sudo add-apt-repository ppa:chris-lea/node.js
9+
* sudo apt-get update
10+
* sudo apt-get install nodejs
11+
12+
Install Dependancies
13+
--------------------
14+
* cd /path/to/folder
15+
* npm install
16+
17+
18+
List of Dependancies
19+
---------------
20+
21+
* [async](https://github.com/caolan/async)
22+
* [async-foreach](https://github.com/cowboy/javascript-sync-async-foreach)
23+
* [express.js](http://expressjs.com/)
24+
* [jquery](http://jquery.com/)
25+
* [socket.io](http://socket.io/)
26+
* [natural](https://github.com/NaturalNode/natural)
27+
* [pos](https://github.com/dariusk/pos-js)
28+
* [js2xmlparser](https://www.npmjs.org/package/js2xmlparser)
29+
* [number-extended](https://github.com/doug-martin/number-extended)
30+
* [request](https://github.com/mikeal/request)
31+
* [mathjs](http://mathjs.org/)
32+
33+
34+
API Overview
35+
-------------
36+
<url>/?q=<query>&eng[engine]=<numOfResults>&resType=<typesOfResult(s)> &keys[apiKey(s)]=&formType=<resultFormat>
37+
38+
###General Rules :
39+
1. Query + Api Keys are mandatory (unless you specify Blekko to search only)
40+
2. Defaults are set to use everything, i.e. if you leave out result type it tries to get everything
41+
3. Be specific in what you want to return and query will be faster
42+
43+
44+
###<url> :
45+
This is the base URL e.g. https://localhost.com/
46+
47+
48+
###<query> :
49+
This is the query must be a html encoded string surronded in double quotes
50+
https://localhost.com/?q=Big Blue
51+
52+
53+
###<engines> :
54+
Each engine must be specified e.g. &eng[goog]&eng[blek]
55+
You can also specify the amount of results to return e.g &eng[goog]=100
56+
Specify the amount of results to return in 10's i.e. 60,70,80
57+
If you choose any other numbers not it will round to closest 10 e.g. &eng[blek]=17 you get 20 back
58+
59+
For Google use goog e.g. &eng[goog]
60+
For Bing use bing e.g. &eng[bing]
61+
For Blekko use blek e.g. &eng[blek]
62+
For all 3 use all e.g &eng[all]
63+
64+
Example:
65+
https://faas.stage1.mybluemix.net/q?=IBM&eng[all]=100 //Gets 100 results from each search engine
66+
https://faas.stage1.mybluemix.net/q?=IBM&eng[all]=60 //Gets 60 results from each engine
67+
https://faas.stage1.mybluemix.net/q?=IBM&eng[goog]=10&eng[bing]=10 //Gets 10 results from google bing
68+
https://faas.stage1.mybluemix.net/q?=IBM&eng[blek]=100 //
69+
70+
Note:
71+
1. Choosing different amounts for each engine e.g. bing[30] and goog[60] may effect aggregation and k-means clustering
72+
2. Choosing to return 100 search results for each engine ie 300 results overall will slow down your query
73+
74+
75+
###<typesOfResult(s)> :
76+
The types of results must be specified next.
77+
These are the different list of results types created from the engines
78+
Each list of the results that were returned by the individual search engines will also be returned
79+
individually this is to specify other methods
80+
81+
For Combination Multiply Non-Zero use combmnz
82+
For Reciprocal Rank Fusion Results use rrf
83+
For k-means Clustering Restuls use kmeans
84+
For a list of related results use related
85+
For all result types to be returned use all
86+
87+
Example:
88+
https://faas.stage1.mybluemix.net/q?=IBM&eng[all]=100resType=all
89+
https://faas.stage1.mybluemix.net/q?=IBM&eng[goog]=100&eng[bing]=100&resType=rrf+kmeans
90+
91+
92+
###<apiKey(s)> :
93+
These are the API keys you have need will need to get before you try a request
94+
If left blank it will throw an error
95+
96+
[Google Custom Search API](https://support.google.com/customsearch/answer/2631040?hl=en)
97+
Set up a free account with your gmail and Googles API console and recieve up to 100 results per day
98+
The amount of results you specify to recieve will have an effect on the amount of total
99+
searches you can make. For example 10 searches asking for 100 results will use up your alotment
100+
When you sign up you will recieve a project number called a CX and and API key
101+
Specify you google API details in the following format
102+
&keys[goog]=CX+APIkey
103+
104+
Bing Search API
105+
Set up a free Bing Seach API account with Microsoft Azure
106+
The amount of results you can make 5000 searches per month.
107+
When you sign up you will recieve an API key
108+
Specify it it in the following format
109+
&keys[bing]=APIkey
110+
111+
Example:
112+
https://faas.stage1.mybluemix.net/q?="IBM"&eng[all]&resType=all&keys[goog]=CX+APIkey&keys[bing]=APIKey
113+
https://faas.stage1.mybluemix.net/q?="IBM"&eng[goog]=100&resType=rrf+kMeans&keys[goog]=CX+APIkey
114+
https://faas.stage1.mybluemix.net/q?="IBM"&eng[blek]&resType=combMNZ /*Key not needed/*
115+
116+
117+
118+
###<resultFormat> :
119+
This is the format of the result you want to be returned
120+
At the momment the API supports the only JSON and XML
121+
but let me know what else you'd like an I'll try to accomadate it
122+
This can be specifed using
123+
* JSON
124+
* XMl
125+
If left blank it defaults to json
126+
127+
Example:
128+
https://faas.stage1.mybluemix.net/q?="IBM"&eng[all]&resType=all&keys[goog]=CX+APIkey&keys[bing]=APIKey&formType=xml
129+
https://faas.stage1.mybluemix.net/q?="IBM"&eng[goog]=100&resType=rrf+kMeans&keys[goog]=CX+APIkey&formType=json
130+
131+
132+
Response
133+
------
134+
135+
Generally if you request the full feature set of this API your response will come in the following format
136+
137+
###Response in JSON
138+
{
139+
msg : "Successful Results",
140+
reqDetails : //An object with the Details of your Request,
141+
goog: //An array of objects with Results from Googles Custom Search API,
142+
blek: //An array of objects with Results from Blekkos Search API,
143+
bing: //An array of objects with Results from Bings Search API,
144+
rrf: //An array of objects from different engines aggregated using Reciprocal Rank Fusion,
145+
combMNZ: //An array of objects from different engines aggregated using Combination Multiply Non-Zero,
146+
kmeans: //An object with 3 arrays of,
147+
time: //The time taken in miliseconds for the search Request,
148+
bingRelated: //An array of Realated searches from the Bing realted search terms api
149+
}
150+
151+
###Resonse in XML
152+
<?xml version="1.0" encoding="UTF-8"?>
153+
<results> //Root Node
154+
<msg>Successful Results</msg>
155+
<goog></goog>
156+
<blek></blek>
157+
<bing></bing>
158+
<rrf></rrf>
159+
<combMNZ></combMNZ>
160+
<kmeans></kmeans>
161+
<time></time>
162+
<bingRelated></bingRelated>
163+
</results>
164+
165+
166+
167+
168+
169+
170+
171+
172+
173+
174+
175+
176+
177+
178+
179+
180+
181+
182+

‎app.js

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
var express = require('express');
2+
var path = require('path');
3+
var favicon = require('static-favicon');
4+
var logger = require('morgan');
5+
var cookieParser = require('cookie-parser');
6+
var bodyParser = require('body-parser');
7+
var httpParser = require("./my_modules/httpParser")
8+
//Increase Socket pooling
9+
require('http').globalAgent.maxSockets = Infinity;
10+
var app = express();
11+
12+
app.use(favicon());
13+
app.use(express.errorHandler());
14+
app.use(logger('dev'));
15+
app.use(express.compress());
16+
app.use(bodyParser.json());
17+
app.use(bodyParser.urlencoded());
18+
app.use(cookieParser());
19+
20+
var express = require('express');
21+
22+
app.all('*', function(req, res){
23+
var date = new Date();//Set start timestamp
24+
httpParser.parseData(req, res, date);
25+
});
26+
27+
/// catch 404 and forwarding to error handler
28+
app.use(function(req, res, next) {
29+
var err = new Error('Not Found');
30+
err.status = 404;
31+
next(err);
32+
});
33+
34+
/// error handlers
35+
36+
// development error handler
37+
// will print stacktrace
38+
if (app.get('env') === 'development') {
39+
app.use(function(err, req, res, next) {
40+
res.status(err.status || 500);
41+
res.render('error', {
42+
message: err.message,
43+
error: err
44+
});
45+
});
46+
}
47+
48+
// production error handler
49+
// no stacktraces leaked to user
50+
app.use(function(err, req, res, next) {
51+
res.status(err.status || 500);
52+
res.render('error', {
53+
message: err.message,
54+
error: {}
55+
});
56+
});
57+
58+
59+
// VCAP_APPLICATION contains useful information about a deployed application.
60+
var appInfo = JSON.parse(process.env.VCAP_APPLICATION || "{}");
61+
// TODO: Get application information and use it in your app.
62+
63+
// VCAP_SERVICES contains all the credentials of services bound to
64+
// this application. For details of its content, please refer to
65+
// the document or sample of each service.
66+
var services = JSON.parse(process.env.VCAP_SERVICES || "{}");
67+
// TODO: Get service credentials and communicate with bluemix services.
68+
69+
// The IP address of the Cloud Foundry DEA (Droplet Execution Agent) that hosts this application:
70+
var host = (process.env.VCAP_APP_HOST || 'localhost');
71+
// The port on the DEA for communication with the application:
72+
var port = (process.env.VCAP_APP_PORT || 3000);
73+
// Start server
74+
app.listen(port, host);
75+
console.log('App started on port ' + port);
76+

‎instructions.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
Get started with faas
2+
-----------------------------------
3+
Welcome to your new Node.js app!
4+
5+
Node.js app runs on a server-side high performance, highly scalable, event-driven environment with non-blocking I/O that is programmed with the JavaScript programming language.
6+
7+
1. [Install the cf command-line tool](https://www.stage1.ng.bluemix.net/docs/redirect.jsp?name=cf-instructions).
8+
2. [Download the starter application package](https://ace.stage1.ng.bluemix.net:443/rest/../rest/apps/8d0bd8ec-9fc8-4980-8240-6bbeea53902d/starter-download)
9+
3. Extract the package and `cd` to it.
10+
4. Connect to BlueMix:
11+
12+
cf api https://api.stage1.ng.bluemix.net
13+
14+
5. Log into BlueMix:
15+
16+
cf login -u [email protected]
17+
cf target -o [email protected] -s dev
18+
19+
6. Deploy your app:
20+
21+
cf push faas
22+
23+
7. Access your app: [faas.stage1.ng.bluemix.net](//faas.stage1.ng.bluemix.net)
24+

‎license.txt

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
The MIT License (MIT)
2+
3+
Copyright (c) 2014 Peter Hanily
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in
13+
all copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21+
THE SOFTWARE.

‎manifest.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
applications:
2+
- disk_quota: 1024M
3+
host: faas
4+
name: faas
5+
command: node app.js
6+
path: .
7+
domain: stage1.mybluemix.net
8+
instances: 1
9+
memory: 1024M

‎my_modules/clustering/assignToClusters.js

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
var forEach = require('async-foreach').forEach;
2+
3+
function toCluster (k, docs, clusters, distMatrix){
4+
//From the distance martix computed from the distance to new centroids
5+
//put documents into the new clusters
6+
forEach(docs,
7+
function(item, index, arr){
8+
var nearest = 1;
9+
var clustToAssign;
10+
for(var i = 0; i < k; i++){
11+
//Assign object to the seed that it is closest too (dist = 0 for doc that is seed no cluster without 1 doc)
12+
if(distMatrix[i][item.position] <= nearest){
13+
nearest = distMatrix[i][item.position];
14+
clustToAssign = i;
15+
}
16+
}
17+
clusters[clustToAssign].push(item);//assign doc to cluster
18+
});
19+
}
20+
21+
exports.toCluster = toCluster;

0 commit comments

Comments
 (0)