Code

Royce branch#109

Code

Open

xKimChip wants to merge 93 commits intoMondego:masterMondego/spacetime-crawler4py:masterfrom 26dre:royce_branch26dre/spacetime-crawler4py:royce_branchCopy head branch name to clipboard

Commits

Commits on Oct 24, 2024

• stuff

  Eucalyptus5

  committed
  351e76b

  Copy full SHA for 351e76b

Commits on Oct 25, 2024

• fixing repetitve web crawling

  Eucalyptus5

  committed
  19a0501

  Copy full SHA for 19a0501

• deleted print statement

  Eucalyptus5

  committed
  cb3d73f

  Copy full SHA for cb3d73f

Commits on Oct 26, 2024

• fixes

  Eucalyptus5

  committed
  1507090

  Copy full SHA for 1507090

• banned some domains

  Eucalyptus5

  committed
  558ec46

  Copy full SHA for 558ec46

Commits on Oct 28, 2024

• Royce ID

  xKimChip

  committed
  1e05806

  Copy full SHA for 1e05806

Commits on Oct 29, 2024

• Added n_grams and some basic multi threading safety for the deque of website token sets, the same pattern can be modeled for other global data structures; changes made have been non destructive in …

  Show description for 7a31b19
  

  Andre Khmelnitsky

  committed
  7a31b19

  Copy full SHA for 7a31b19

Commits on Oct 31, 2024

• REGAE

  xKimChip

  committed
  b4b7c8c

  Copy full SHA for b4b7c8c

• added the trie and some extra stuff for safety purposes

  

  Andre Khmelnitsky

  committed
  a1c9f98

  Copy full SHA for a1c9f98

• Alphabet

  xKimChip

  committed
  d43afab

  Copy full SHA for d43afab

• added some basic renaming because of new type alias

  

  Andre Khmelnitsky

  committed
  d03dad6

  Copy full SHA for d03dad6

• follows last changes

  

  Andre Khmelnitsky

  committed
  6d7860c

  Copy full SHA for 6d7860c

• Fixed regex paranthesis

  xKimChip

  committed
  a9f22c1

  Copy full SHA for a9f22c1

• added the globals to their own file to make mutexing them way easier

  

  Andre Khmelnitsky

  committed
  91ad04a

  Copy full SHA for 91ad04a

• Merge branch 'master' into andre_branch

  26dre

  authored
  4b0d185

  Copy full SHA for 4b0d185

• Merge pull request #1 from 26dre/andre_branch

  Show description for ceaa9ef
  26dre

  authored
  ceaa9ef

  Copy full SHA for ceaa9ef

• put the domain trie prints in the top level module so it does not run on import

  

  Andre Khmelnitsky

  committed
  20ed00f

  Copy full SHA for 20ed00f

• Merge branch 'royce_branch' of https://github.com/26dre/spacetime-crawler4py into royce_branch

  xKimChip

  committed
  361592e

  Copy full SHA for 361592e

• added a new global variable URL_THRESHOLD_SIMILARITY that checks against whether or not the URL is similar or not and whether or not to evaluate a url based on the path similarity

  

  Andre Khmelnitsky

  committed
  1cc774a

  Copy full SHA for 1cc774a

• Merge pull request #2 from 26dre/andre_branch

  Show description for a5538ca
  26dre

  authored
  a5538ca

  Copy full SHA for a5538ca

Commits on Nov 1, 2024

• added a basic test to link similarity.py as well as a test suite function that can be used anywhere and should likely be extracted out into its own separate file for testing

  

  Andre Khmelnitsky

  committed
  9eac990

  Copy full SHA for 9eac990

Commits on Nov 2, 2024

• added a test suite class and test cases for link_similarity.py to confirm that it works, changed global url similarity to a higher threshold of .85 instead of .8

  

  Andre Khmelnitsky

  committed
  6300977

  Copy full SHA for 6300977

• Merge pull request #3 from 26dre/andre_branch

  Show description for 922f659
  26dre

  authored
  922f659

  Copy full SHA for 922f659

• separated tokenizer

  dillonct

  committed
  0bac6ec

  Copy full SHA for 0bac6ec

• fixed id

  dillonct

  committed
  710448f

  Copy full SHA for 710448f

• Merge pull request #4 from 26dre/dillon_branch

  Show description for 8c0a954
  dillonct

  authored
  8c0a954

  Copy full SHA for 8c0a954

• Merge remote-tracking branch 'origin' into royce_branch

  xKimChip

  committed
  7ed6ad2

  Copy full SHA for 7ed6ad2

• put the ngrams in their own module to allow for easier code readability

  

  Andre Khmelnitsky

  committed
  ed90d08

  Copy full SHA for ed90d08

• Merge pull request #5 from 26dre/andre_branch

  Show description for acab58e
  26dre

  authored
  acab58e

  Copy full SHA for acab58e

• moved around all n_grams things to its own ngrams.py file in order to make easier and more logical to read the code base, will have to add changes in ngrams.py to allow for the reading and writing …

  Show description for e9708c9
  

  Andre Khmelnitsky

  committed
  e9708c9

  Copy full SHA for e9708c9

• Merge pull request #6 from 26dre/andre_branch

  Show description for 29f0b78
  26dre

  authored
  29f0b78

  Copy full SHA for 29f0b78

• Merge remote-tracking branch 'origin' into royce_branch

  xKimChip

  committed
  780ccd4

  Copy full SHA for 780ccd4

• Merge remote-tracking branch 'origin' into royce_branch

  xKimChip

  committed
  fe7b5d7

  Copy full SHA for fe7b5d7

• Merge branch 'master' of https://github.com/26dre/spacetime-crawler4py

  xKimChip

  committed
  b0222ae

  Copy full SHA for b0222ae

• small basic changes to globals and link similarity for readability

  

  Andre Khmelnitsky

  committed
  5dc73d9

  Copy full SHA for 5dc73d9

• Dates again

  xKimChip

  committed
  87e9eb3

  Copy full SHA for 87e9eb3

• local_regex.py

  xKimChip

  committed
  1abfdd9

  Copy full SHA for 1abfdd9

• Merge branch 'royce_branch' of https://github.com/26dre/spacetime-crawler4py into royce_branch

  xKimChip

  committed
  b2ed09d

  Copy full SHA for b2ed09d

• changed type alias and tested

  dillonct

  committed
  62fb83b

  Copy full SHA for 62fb83b

• Merge pull request #7 from 26dre/dillon_branch

  Show description for 1defa50
  dillonct

  authored
  1defa50

  Copy full SHA for 1defa50

• added some function to read and write to global variables in a more efficient way, should be my last commit for the day

  

  Andre Khmelnitsky

  committed
  9a87240

  Copy full SHA for 9a87240

• Merge pull request #8 from 26dre/andre_branch

  Show description for 5745657
  26dre

  authored
  5745657

  Copy full SHA for 5745657

• andre id

  xKimChip

  committed
  d6f8c26

  Copy full SHA for d6f8c26

• Runs, but my laptop crashed

  xKimChip

  committed
  72d3918

  Copy full SHA for 72d3918

Commits on Nov 3, 2024

• added some basic thread safety, functions work in theory but have not been tested with multiple threads. includes and uses the interface for going through function as defined in globals.py read and…

  Show description for 9b6d956
  

  Andre Khmelnitsky

  committed
  9b6d956

  Copy full SHA for 9b6d956

• Merge pull request #9 from 26dre/andre_branch

  Show description for 8e5030e
  26dre

  authored
  8e5030e

  Copy full SHA for 8e5030e

• added a non destructive check on whether the url should be evaluated in the first place earlier in a more logical and concise way than earlier implemenation, no real change made

  

  Andre Khmelnitsky

  committed
  5b59bb0

  Copy full SHA for 5b59bb0

• Merge pull request #10 from 26dre/andre_branch

  Show description for ac287fd
  26dre

  authored
  ac287fd

  Copy full SHA for ac287fd

Commits on Nov 5, 2024

• added stuff to domainTrie in order to make it more usable and allow the domain tree to report back the number of new urls found, made a multithread safe funciton in globals.py that accesses the glo…

  Show description for b848495
  

  Andre Khmelnitsky

  committed
  b848495

  Copy full SHA for b848495

• Merge pull request #11 from 26dre/andre_branch

  Show description for 0e6c35a
  26dre

  authored
  0e6c35a

  Copy full SHA for 0e6c35a

• made a couple more data structures thread_safe, moved some functions from scraper into globals.py for consistency and then added threadsafe versions to it

  

  Andre Khmelnitsky

  committed
  28c0c89

  Copy full SHA for 28c0c89

• can save files now

  dillonct

  committed
  d5fb4f4

  Copy full SHA for d5fb4f4

• Merge pull request #12 from 26dre/andre_branch

  Show description for 2da4137
  26dre

  authored
  2da4137

  Copy full SHA for 2da4137

• fixed to globals

  dillonct

  committed
  5ea8ad3

  Copy full SHA for 5ea8ad3

• Merge branch 'master' into dillon

  dillonct

  authored
  eff0a93

  Copy full SHA for eff0a93

• Merge pull request #13 from 26dre/dillon

  Show description for 46b03bf
  dillonct

  authored
  46b03bf

  Copy full SHA for 46b03bf

• added some basic things to ngrams and some small adjustments to scraper

  

  Andre Khmelnitsky

  committed
  2fae8f3

  Copy full SHA for 2fae8f3

• Merge pull request #14 from 26dre/andre_branch

  Show description for 6e3652c
  26dre

  authored
  6e3652c

  Copy full SHA for 6e3652c

• small update to globals

  

  Andre Khmelnitsky

  committed
  44c357d

  Copy full SHA for 44c357d

• Merge pull request #15 from 26dre/andre_branch

  Show description for c87201a
  26dre

  authored
  c87201a

  Copy full SHA for c87201a

• updated word frequency logging

  dillonct

  committed
  009fc59

  Copy full SHA for 009fc59

• Merge pull request #16 from 26dre/dillon

  Show description for 0483a04
  dillonct

  authored
  0483a04

  Copy full SHA for 0483a04

• update logging

  dillonct

  committed
  aa2235e

  Copy full SHA for aa2235e

• Merge pull request #17 from 26dre/dillon

  Show description for ebbc977
  dillonct

  authored
  ebbc977

  Copy full SHA for ebbc977

• andre stop writing python code like C and updated logging

  dillonct

  committed
  4a23d9a

  Copy full SHA for 4a23d9a

Commits on Nov 9, 2024

• fixed ngrams turns out i was doing doing some wonky stuff with additions and the like, the testing was done in the main and can be uncommeneted, two additinoal files are needed for testing and ther…

  Show description for 5b66456
  

  Andre Khmelnitsky

  committed
  5b66456

  Copy full SHA for 5b66456

• Merge pull request #18 from 26dre/andre_branch

  Show description for d126ef7
  26dre

  authored
  d126ef7

  Copy full SHA for d126ef7

Commits on Nov 11, 2024

• added some extra safety checks and made everything accessible via multithreading

  

  Andre Khmelnitsky

  committed
  196c447

  Copy full SHA for 196c447

• Merge pull request #19 from 26dre/andre_branch

  Show description for 68028b5
  26dre

  authored
  68028b5

  Copy full SHA for 68028b5

• added some basic changes to the scraper so that it works thru more regex

  

  Andre Khmelnitsky

  committed
  19463ab

  Copy full SHA for 19463ab

• Merge pull request #20 from 26dre/andre_branch

  Show description for d8adb93
  26dre

  authored
  d8adb93

  Copy full SHA for d8adb93

• added one extra safety check inside of the worker

  

  Andre Khmelnitsky

  committed
  f29af3a

  Copy full SHA for f29af3a

• Merge pull request #21 from 26dre/andre_branch

  Show description for e8c51d1
  26dre

  authored
  e8c51d1

  Copy full SHA for e8c51d1

• changed worker so that it should work multithreaded with politeness by checking to make sure that the urls thtat it gets are not currently already being looked at at the same time

  

  Andre Khmelnitsky

  committed
  e9c8229

  Copy full SHA for e9c8229

• added up to 20 threads as it should be threadsafe now

  

  Andre Khmelnitsky

  committed
  d892c7f

  Copy full SHA for d892c7f

• Merge pull request #22 from 26dre/andre_branch

  Show description for 18fc22b
  26dre

  authored
  18fc22b

  Copy full SHA for 18fc22b

Commits on Nov 12, 2024

• combined a couple of if statements with same early return for the sake of readability. Also found the < operator that was breaking the code so provided the fix

  

  Andre Khmelnitsky

  committed
  3abcc12

  Copy full SHA for 3abcc12

• made some changes in globals.py so that it is made more readable now, no other changes amde

  

  Andre Khmelnitsky

  committed
  0a19d12

  Copy full SHA for 0a19d12

• made the n grams code fully threadsafe (it wasn't before) just utilizes a function with the same body but then utilizes a lock for thread safety

  

  Andre Khmelnitsky

  committed
  c66e729

  Copy full SHA for c66e729

• Merge pull request #23 from 26dre/andre_branch

  Show description for a1afef8
  26dre

  authored
  a1afef8

  Copy full SHA for a1afef8

Commits on Nov 13, 2024

• M1 kinda complete

  Eucalyptus5

  committed
  14fb443

  Copy full SHA for 14fb443

Commits on Nov 16, 2024

• added query retrieval (m2), is now mostly complete (but not totally, requires creating the main funciton but almost everything is alreday place such that it can work

  

  Andre Khmelnitsky

  committed
  7e01969

  Copy full SHA for 7e01969

Commits on Nov 17, 2024

• fixed the query input, now correctly takes in queries that include AND or OR (no support yet for NOT)

  

  Andre Khmelnitsky

  committed
  4da5927

  Copy full SHA for 4da5927

• added small change that allows globals to compile on openlab

  

  Andre Khmelnitsky

  committed
  f2965c2

  Copy full SHA for f2965c2

• added changes to the postings class that allow it to be hashed and used by sets and the like, no functinoal changes made to the code of the class, also makes the pickled file a global variable

  

  Andre Khmelnitsky

  committed
  c6f7a45

  Copy full SHA for c6f7a45

• added changes that allow OR and AND queries to work in a multithreaded environment

  

  Andre Khmelnitsky

  committed
  7c32cbb

  Copy full SHA for 7c32cbb

• fully tested against the large .pkl document produced

  

  Andre Khmelnitsky

  committed
  cd1806e

  Copy full SHA for cd1806e

• Merge pull request #24 from 26dre/andre_branch_index_cons

  Show description for 103f13b
  26dre

  authored
  103f13b

  Copy full SHA for 103f13b

Commits on Nov 25, 2024

• added a testing suite and file for testing

  

  Andre Khmelnitsky

  committed
  b5b0ea0

  Copy full SHA for b5b0ea0

Commits on Nov 26, 2024

• Lemmatize

  xKimChip

  committed
  5183f9f

  Copy full SHA for 5183f9f

• Index of Index

  xKimChip

  committed
  38af34d

  Copy full SHA for 38af34d

• Starting Index of Index

  xKimChip

  committed
  0ce67bc

  Copy full SHA for 0ce67bc

Commits on Nov 30, 2024

• Added weights to term frequencies. I may change it to addition instead of a multiplier.

  xKimChip

  committed
  0b23abe

  Copy full SHA for 0b23abe