rmgr
|
efe6dea1f5
|
Fix crawling. Add initial linksfile crawling. Still need to remove records as they are processed.
|
2024-01-01 20:52:12 +10:30 |
|
rmgr
|
f4ea8ad1d7
|
Respect robots.txt
|
2024-01-01 19:53:22 +10:30 |
|
rmgr
|
b43343e0ee
|
Fix recursive crawling.
|
2023-12-12 17:35:15 +10:30 |
|
rmgr
|
21961fced6
|
Add notes to index.py
|
2023-12-06 08:31:54 +10:30 |
|
rmgr
|
9fc2e1af53
|
Implement recursive page crawling
|
2023-12-06 08:29:39 +10:30 |
|
rmgr
|
3d7b72e5ef
|
Join counts for multiple words
|
2023-11-30 17:26:59 +10:30 |
|
rmgr
|
d30397cefa
|
Add count of times word appears on a site to index.
|
2023-11-30 08:03:43 +10:30 |
|
rmgr
|
f36ab2fbfb
|
Initial commit
|
2023-11-28 20:51:54 +10:30 |
|