rmgr/search-engine
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity

Make excluded file types more robust

Browse source
This commit is contained in:
rmgr 2024-06-08 20:24:21 +09:30
parent 98efe9d1a2
commit e3c67b64e6
2 changed files with 9 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

2
todo
View file

@ -6,4 +6,6 @@
[x] Add clustered index to document_ngrams table model
[x] Add clustered index to document_tokens table model
[ ] Add ddl command to create partition tables
[ ] Investigate whether or not robots.txt is as aggressive as I'm making ito ut to be
[ ] Instead of starting from a random page on the site, go to root and find site map and crawl that