bMorgan/link-crawler
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

done

Browse source
This commit is contained in:
bMorgan01 2022-09-19 19:26:35 -06:00
parent 75c911f6d2
commit 7814e210bc
2 changed files with 53 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

6
crawl.conf
View file

@ -1,7 +1,5 @@
# Domain Ex: benrmorgan.com
benrmorgan.com
# Prefix Ex: http://www.
http://www.
# Target Ex: http://www.google.com
http://dev-builder.benrmorgan.com
# Ignore urls containing Ex: /files/
/files/
/images/