link-crawler/crawl.conf
bMorgan01 7814e210bc done
2022-09-19 19:26:35 -06:00

5 lines
131 B
Text

# Target Ex: http://www.google.com
http://dev-builder.benrmorgan.com
# Ignore urls containing Ex: /files/
/files/
/images/