Skip to content

Commit 485a187

Browse files
author
baruch
authoredJun 26, 2019
Add files via upload
lab for MongoDB
1 parent 7472864 commit 485a187

8 files changed

+26023
-0
lines changed
 

‎Install_mongodb.md

+131
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# How to install MongoDB on Linux
2+
3+
- change the ulimit for this user as
4+
in /etc/security/limits.conf or if using the service in the service file :
5+
```
6+
(file size): unlimited
7+
(cpu time): unlimited
8+
(virtual memory): unlimited [1]
9+
(open files): 64000
10+
(memory size): unlimited [1] [2]
11+
(processes/threads): 64000
12+
13+
```
14+
15+
## radhet
16+
```bash
17+
vi /etc/yum.repos.d/mongodb-org-4.0.repo
18+
```
19+
[mongodb-org-4.0]
20+
name=MongoDB Repository
21+
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/
22+
gpgcheck=1
23+
enabled=1
24+
gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc
25+
```
26+
27+
```bash
28+
sudo yum install -y mongodb-org
29+
```
30+
31+
## deb
32+
33+
https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian/
34+
35+
36+
## Run mongodb
37+
38+
- this is a small test on my laptop so is will run MongoDB stand-alone
39+
- in the config yaml i change the log path and db path :
40+
- i run --oplogSize 10 reduce the disk space
41+
42+
43+
- if we need to change the data location
44+
```
45+
mkdir --p /u01/data/mongo01
46+
mkdir --p /u01/log/mongo01
47+
mkdir --p /u01/config/mongo01
48+
49+
cp /etc/mongod.conf /u01/config/mongo01
50+
```
51+
- or just create the file :
52+
```yaml
53+
# mongod.conf
54+
55+
# for documentation of all options, see:
56+
# http://docs.mongodb.org/manual/reference/configuration-options/
57+
58+
# where to write logging data.
59+
systemLog:
60+
destination: file
61+
logAppend: true
62+
path: /u01/log/mongo01/mongod.log
63+
64+
# Where and how to store data.
65+
storage:
66+
dbPath: /u01/data/mongo01
67+
journal:
68+
enabled: true
69+
# engine:
70+
# wiredTiger:
71+
72+
# how the process runs
73+
processManagement:
74+
fork: true # fork and run in background
75+
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
76+
timeZoneInfo: /usr/share/zoneinfo
77+
78+
# network interfaces
79+
net:
80+
port: 27017
81+
bindIp: 127.0.0.1 # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
82+
83+
84+
#security:
85+
86+
#operationProfiling:
87+
88+
#replication:
89+
90+
#sharding:
91+
92+
## Enterprise-Only Options
93+
94+
#auditLog:
95+
96+
#snmp:
97+
98+
99+
```
100+
101+
102+
103+
## enable service
104+
105+
```
106+
sudo chkconfig mongod on
107+
```
108+
## start mongodb
109+
110+
```
111+
sudo service mongod start
112+
```
113+
- or from command line
114+
```
115+
mongod --config /u01/config/mongo01/mongod.conf --oplogSize 10 --fork
116+
```
117+
118+
119+
```bash
120+
ps aux |grep mongo
121+
vagrant 3499 0.8 1.0 1003384 58692 ? Sl 20:00 0:00 mongod --config /u01/config/mongo01/mongod.conf --oplogSize 10 --fork
122+
```
123+
124+
```bash
125+
echo "db.version()" |mongo
126+
MongoDB shell version v3.6.1
127+
connecting to: mongodb://127.0.0.1:27017
128+
MongoDB server version: 3.6.1
129+
3.6.1
130+
bye
131+
```

‎agreation.md

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
2+
# aggregation
3+
4+
5+
```sh
6+
ps -ef |grep mongo |grep -v grep |awk '{print "kill " $2}'
7+
rm -rf /u01/data/mongo01
8+
mkdir --p /u01/data/mongo01
9+
10+
mongod --dbpath /u01/data/mongo01 --logpath /u01/data/mongo01/log.log --logappend --oplogSize 10 --smallfiles --fork
11+
```
12+
## import data
13+
```bash
14+
## wget http://media.mongodb.org/zips.json
15+
mongoimport --drop -d pcat -c zipcodes /u01/demo/zips.json
16+
17+
18+
```
19+
## Return States with Populations above 10 Million
20+
* in sql
21+
```sql
22+
SELECT state, SUM(pop) AS totalPop
23+
FROM zipcodes
24+
GROUP BY state
25+
HAVING totalPop >= (10*1000*1000)
26+
```
27+
28+
* In this example, the aggregation pipeline consists of the $group stage followed by the $match stage:
29+
30+
```js
31+
32+
use pcat
33+
db.zipcodes.aggregate( [
34+
{ $group: { _id: "$state", totalPop: { $sum: "$pop" } } },
35+
{ $match: { totalPop: { $gte: 10*1000*1000 } } }
36+
] )
37+
```
38+
39+
## Return Largest and Smallest Cities by State
40+
41+
```js
42+
use pcat
43+
44+
db.zipcodes.aggregate( [
45+
{ $group:
46+
{
47+
_id: { state: "$state", city: "$city" },
48+
pop: { $sum: "$pop" }
49+
}
50+
},
51+
{ $sort: { pop: 1 } },
52+
{ $group:
53+
{
54+
_id : "$_id.state",
55+
biggestCity: { $last: "$_id.city" },
56+
biggestPop: { $last: "$pop" },
57+
smallestCity: { $first: "$_id.city" },
58+
smallestPop: { $first: "$pop" }
59+
}
60+
},
61+
62+
// the following $project is optional, and
63+
// modifies the output format.
64+
65+
{ $project:
66+
{ _id: 0,
67+
state: "$_id",
68+
biggestCity: { name: "$biggestCity", pop: "$biggestPop" },
69+
smallestCity: { name: "$smallestCity", pop: "$smallestPop" }
70+
}
71+
}
72+
] )
73+
74+
```

‎change_streams.md

+127
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
2+
# aggregation
3+
4+
## Run Replication MongoDB
5+
** The $changeStream stage is only supported on replica sets **
6+
7+
starting with 3 nodes 1 arbiter 2 data nodes
8+
9+
10+
11+
## clean
12+
```sh
13+
# clean
14+
ps -ef |grep mongo |grep -v grep |awk '{print "kill " $2}'
15+
rm -rf /u01/data
16+
17+
18+
# create
19+
mkdir -p /u01/data/t{1..4}
20+
```
21+
22+
23+
## run mongod
24+
```sh
25+
mongod --port 27001 --replSet t --dbpath /u01/data/t1 --logpath /u01/data/t1/log.log --logappend --oplogSize 10 --smallfiles --fork
26+
mongod --port 27002 --replSet t --dbpath /u01/data/t2 --logpath /u01/data/t2/log.log --logappend --oplogSize 10 --smallfiles --fork
27+
mongod --port 27003 --replSet t --dbpath /u01/data/t3 --logpath /u01/data/t3/log.log --logappend --oplogSize 10 --smallfiles --fork
28+
mongod --port 27004 --replSet t --dbpath /u01/data/t4 --logpath /u01/data/t4/log.log --logappend --oplogSize 10 --smallfiles --fork
29+
30+
```
31+
32+
33+
## create replica set
34+
```
35+
mongo host76:27001
36+
```
37+
38+
39+
```js
40+
//create replica config
41+
42+
cfg = {
43+
_id: "t",
44+
members : [
45+
{_id:0,host:"host76:27001","priority": 90},
46+
{_id:1,host:"host76:27002","arbiterOnly" : true },
47+
{_id:2,host:"host76:27003","priority": 30}
48+
]
49+
}
50+
51+
```
52+
```js
53+
rs.initiate(cfg)
54+
```
55+
56+
57+
## lets create stock price stream
58+
```js
59+
60+
use test
61+
var docs = [
62+
{ ticker: "WIX", price: 140 },
63+
{ ticker: "WIX", price: 150 },
64+
{ ticker: "WIX", price: 160 },
65+
{ ticker: "WIX", price: 130 },
66+
{ ticker: "WIX", price: 150 }
67+
];
68+
db.Stocks.insert(docs)
69+
```
70+
71+
72+
## install monog on local node
73+
```
74+
mkdir mongo-proj && cd mongo-proj
75+
npm init -y
76+
npm install mongodb --save
77+
```
78+
79+
## js code
80+
81+
index.js
82+
```js
83+
const mongo = require("mongodb").MongoClient;
84+
mongo.connect("mongodb://localhost:27001/?replicaSet=t").then(client => {
85+
console.log("Connected to MongoDB server");
86+
// Select DB and Collection
87+
const db = client.db("test");
88+
const collection = db.collection("Stocks");
89+
pipeline = [
90+
{
91+
$match: { "fullDocument.price": { $gte: 250 } }
92+
}
93+
];
94+
// Define change stream
95+
const changeStream = collection.watch(pipeline);
96+
// start listen to changes
97+
changeStream.on("change", function(event) {
98+
console.log(JSON.stringify(event));
99+
});
100+
});
101+
102+
```
103+
104+
* Note
105+
you can change the value of operationType parameter with following operations to listen for different types of changes in a collection:
106+
107+
* Insert
108+
* Replace (Except unique Id)
109+
* Update
110+
* Delete
111+
* Invalidate (Whenever Mongo returns invalid cursor)
112+
113+
114+
115+
## run
116+
117+
```
118+
node index.js
119+
```
120+
121+
122+
## insert
123+
mongo host76:27001
124+
125+
db.Stocks.insert({ ticker: WIX, price: 100 })
126+
127+
db.Stocks.insert({ ticker: WIX, price: 280 })

‎explain.md

+79
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
2+
# explain
3+
4+
5+
## first load data
6+
7+
```bash
8+
ps -ef |grep mongo |grep -v grep |awk '{print "kill " $2}'
9+
rm -rf /u01/data/mongo01
10+
mkdir --p /u01/data/mongo01
11+
12+
mongod --dbpath /u01/data/mongo01 --logpath /u01/data/mongo01/log.log --logappend --oplogSize 10 --smallfiles --fork
13+
```
14+
15+
16+
```bash
17+
mongoimport --drop -d restaurants -c restaurants /u01/demo/restaurants.data
18+
```
19+
20+
## run explain
21+
22+
```js
23+
db.restaurants.explain("executionStats").find(
24+
{ "cuisine": 1, "borough": "Brooklyn"}
25+
);
26+
```
27+
```json
28+
29+
"executionStats" : {
30+
"executionSuccess" : true,
31+
"nReturned" : 0,
32+
"executionTimeMillis" : 10,
33+
"totalKeysExamined" : 0,
34+
"totalDocsExamined" : 25359,
35+
"executionStages" : {
36+
"stage" : "COLLSCAN",
37+
"filter" : {
38+
39+
```
40+
## create index
41+
42+
```js
43+
db.restaurants.createIndex( { cuisine: 1 ,borough: 1} )
44+
45+
46+
```
47+
```js
48+
db.restaurants.explain("executionStats").find(
49+
{ "cuisine": 1, "borough": "Brooklyn"}
50+
);
51+
```
52+
53+
```json
54+
55+
"executionStats" : {
56+
"executionSuccess" : true,
57+
"nReturned" : 0,
58+
"executionTimeMillis" : 0,
59+
"totalKeysExamined" : 0,
60+
"totalDocsExamined" : 0,
61+
"executionStages" : {
62+
"stage" : "FETCH",
63+
"nReturned" : 0,
64+
"executionTimeMillisEstimate" : 0,
65+
...
66+
"invalidates" : 0,
67+
"keyPattern" : {
68+
"cuisine" : 1,
69+
"borough" : 1
70+
},
71+
"indexName" : "cuisine_1_borough_1",
72+
"isMultiKey" : false,
73+
"multiKeyPaths" : {
74+
"cuisine" : [ ],
75+
"borough" : [ ]
76+
},
77+
78+
79+
```

‎restaurants.data

+25,359
Large diffs are not rendered by default.

‎run_cluster.md

+154
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
2+
# Run Cluster
3+
2 shared X 3 replicaset (you have to have more the 2 or 2 data and 1 addArb... )
4+
5+
6+
* only shard the big collections
7+
* pre-split if bulk loading
8+
* pick shard key with care, they aren't easily changeable
9+
* be cognizant of monotonically increasing shard keys
10+
* adding a shard is easy but takes time
11+
* use logical server names for config servers
12+
* don't directly talk to anything except mongos except for dba work
13+
* keep non-mongos processes off of 27017 to avoid mistakes
14+
15+
16+
#clean
17+
ps -ef |grep mongo |grep -v grep |awk '{print "kill " $2}'
18+
19+
rm -rf /u01/data/mongocluster
20+
21+
22+
#
23+
```sh
24+
# clean and create
25+
ps -ef |grep mongo |grep -v grep |awk '{print "kill " $2}'
26+
27+
rm -rf /u01/data/mongocluster && mkdir -p /u01/data/mongocluster/a{0..2} /u01/data/mongocluster/cfg{0..2} /u01/data/mongocluster/b{0..2} /u01/data/mongocluster/s{0..2}
28+
29+
## config nodes (defult port is 27019)
30+
mongod --configsvr --replSet config --port 26050 --dbpath /u01/data/mongocluster/cfg0 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/cfg2/log.log --logappend --fork
31+
32+
mongod --configsvr --replSet config --port 26051 --dbpath /u01/data/mongocluster/cfg1 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/cfg2/log.log --logappend --fork
33+
34+
mongod --configsvr --replSet config --port 26052 --dbpath /u01/data/mongocluster/cfg2 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/cfg2/log.log --logappend --fork
35+
36+
37+
## shard serverd data nodes (2718)
38+
mongod --shardsvr --replSet a --port 27000 --dbpath /u01/data/mongocluster/a0 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/a0/log.log --logappend --fork
39+
mongod --shardsvr --replSet a --port 27001 --dbpath /u01/data/mongocluster/a1 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/a1/log.log --logappend --fork
40+
mongod --shardsvr --replSet a --port 27002 --dbpath /u01/data/mongocluster/a2 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/a2/log.log --logappend --fork
41+
#
42+
mongod --shardsvr --replSet b --port 27010 --dbpath /u01/data/mongocluster/b0 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/b0/log.log --logappend --fork
43+
mongod --shardsvr --replSet b --port 27011 --dbpath /u01/data/mongocluster/b1 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/b1/log.log --logappend --fork
44+
mongod --shardsvr --replSet b --port 27012 --dbpath /u01/data/mongocluster/b2 --smallfiles --oplogSize 50 --logpath /u01/data/mongocluster/b2/log.log --logappend --fork
45+
46+
47+
## mongos (defult 27017)
48+
49+
mongos --configdb "config/host76:26050,host76:26051,host76:26052" --logpath /u01/data/mongocluster/s0/log.log --logappend --fork
50+
51+
52+
mongo #conneto to mongos
53+
```
54+
55+
#
56+
57+
58+
59+
60+
61+
```sh
62+
mongo --port 26050
63+
```
64+
65+
```js
66+
67+
rs.status()
68+
69+
70+
cfg = {
71+
_id: "config",
72+
configsvr: true,
73+
members : [
74+
{_id:0,host:"host76:26050","priority": 100},
75+
{_id:1,host:"host76:26051","priority":100},
76+
{_id:2,host:"host76:26052","priority": 100}
77+
]
78+
}
79+
rs.initiate(cfg)
80+
81+
//connect to the first replica set mongo
82+
mongo --port 27000
83+
rs.status()
84+
85+
///then add more rs
86+
cfg = {
87+
_id: "a",
88+
members : [
89+
{_id:0,host:"host76:27000","priority": 100},
90+
{_id:1,host:"host76:27001","priority":90},
91+
{_id:2,host:"host76:27002","priority": 80}
92+
]
93+
}
94+
rs.initiate(cfg)
95+
96+
```
97+
//
98+
```sh
99+
mongo --port 27010
100+
```
101+
```js
102+
103+
rs.status()
104+
105+
106+
cfg = {
107+
_id: "b",
108+
members : [
109+
{_id:0,host:"host76:27010","priority": 100},
110+
{_id:1,host:"host76:27011","priority":90},
111+
{_id:2,host:"host76:27012","priority": 80}
112+
]
113+
}
114+
rs.initiate(cfg)
115+
116+
```
117+
```
118+
mongo
119+
```
120+
## build shards
121+
122+
```js
123+
/// run mongo // will connet to mongos
124+
/// add shard
125+
//sh.addShard("<relica set name>/<one of the hostnames for the replica>")
126+
sh.addShard("a/host76:27001")
127+
sh.addShard("b/host76:27010")
128+
sh.status()
129+
130+
131+
//The default chunk size for a sharded cluster is 64 megabytes
132+
// for this test lets move it to 1M. (in prod pls keep it 64 megabytes)
133+
use config
134+
db.settings.save( { _id:"chunksize", value: 1 } )
135+
136+
137+
/// create db and data
138+
use testdb
139+
t = db.foo
140+
141+
///
142+
sh.enableSharding("testdb")
143+
sh.shardCollection("testdb.foo",{_id:1},true)
144+
145+
for(var i = 0 ;i < 200000 ;i++){
146+
t.insert({x:i,y:3,x:"test"})
147+
}
148+
//enable sharing on db
149+
150+
sh.status()
151+
t.getShardDistribution()
152+
sh.isBalancerRunning()
153+
154+
```

‎run_one_mongo.md

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
2+
# run one mongodb instance
3+
4+
```sh
5+
ps -ef |grep mongo |grep -v grep |awk '{print "kill " $2}'
6+
rm -rf /u01/data/mongo01
7+
mkdir --p /u01/data/mongo01
8+
9+
mongod --dbpath /u01/data/mongo01 --logpath /u01/data/mongo01/log.log --logappend --oplogSize 10 --smallfiles --fork
10+
11+
12+
mongoimport --drop -d pcat -c testdb /u01/demo/zips.json
13+
```
14+

‎run_replication.md

+85
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Run Replication MongoDB
2+
starting with 3 nodes 1 arbiter 2 data nodes
3+
4+
5+
6+
## clean
7+
```sh
8+
# clean
9+
ps -ef |grep mongo |grep -v grep |awk '{print "kill " $2}'
10+
rm -rf /u01/data
11+
12+
13+
# create
14+
mkdir -p /u01/data/t{1..4}
15+
```
16+
17+
18+
## run mongod
19+
```sh
20+
mongod --port 27001 --replSet t --dbpath /u01/data/t1 --logpath /u01/data/t1/log.log --logappend --oplogSize 10 --smallfiles --fork
21+
mongod --port 27002 --replSet t --dbpath /u01/data/t2 --logpath /u01/data/t2/log.log --logappend --oplogSize 10 --smallfiles --fork
22+
mongod --port 27003 --replSet t --dbpath /u01/data/t3 --logpath /u01/data/t3/log.log --logappend --oplogSize 10 --smallfiles --fork
23+
mongod --port 27004 --replSet t --dbpath /u01/data/t4 --logpath /u01/data/t4/log.log --logappend --oplogSize 10 --smallfiles --fork
24+
25+
```
26+
27+
28+
## create replica set
29+
```
30+
mongo host76:27001
31+
```
32+
33+
34+
```js
35+
//create replica config
36+
37+
cfg = {
38+
_id: "t",
39+
members : [
40+
{_id:0,host:"host76:27001","priority": 90},
41+
{_id:1,host:"host76:27002","arbiterOnly" : true },
42+
{_id:2,host:"host76:27003","priority": 30}
43+
]
44+
}
45+
46+
```
47+
48+
```js
49+
rs.initiate(cfg)
50+
51+
```
52+
53+
```js
54+
// add node
55+
rs.add( { "host": "host76:27004", "priority": 90 ,"buildIndexes" : true,
56+
"tags" : {
57+
"use" : "prod",
58+
"host" : "host76:27004",
59+
"dc" : "primary"
60+
} } )
61+
62+
```
63+
64+
```js
65+
//rs.help()
66+
67+
rs.initiate(cfg)
68+
rs.status()
69+
rs.reconfig(...)
70+
```
71+
## reconfig
72+
```js
73+
///prevent 27003 from primary
74+
cfg = rs.conf()
75+
cfg.members[2].priority = 0
76+
rs.reconfig(cfg)
77+
78+
///move 27004 to primary
79+
cfg = rs.conf()
80+
cfg.members[3].priority = 100
81+
rs.reconfig(cfg)
82+
83+
84+
85+
```

0 commit comments

Comments
 (0)
Please sign in to comment.