link-crawler/crawl.conf
2022-09-25 09:58:13 -06:00

5 lines
123 B
Text

# Target Ex: http://www.google.com
http://www.benrmorgan.com
# Ignore urls containing Ex: /files/
/files/
/images/