sitemap-crawler/crawl.conf
2022-03-29 19:13:27 -06:00

7 lines
No EOL
174 B
Text

# Domain Ex: benrmorgan.com
benrmorgan.com
# Prefix Ex: http://www.
http://www.
# Ignore urls containing Ex: /files/
/files/
/images/