1 Topic by Urdna

  • Urdna
  • Member
  • Offline
  • Registered: 2009-06-13
  • Posts: 20

Topic: robots.txt

When I visit /robots.txt of any of my apps I get the message

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # User-agent: * Disallow: /

Whatever returns that page runs before my app so I can't change that result at all.

Would it be possible for us to be able to make different robots.txt files for our apps?


Thanks,
Urdna

2 Reply by dvbportal

  • dvbportal
  • Member
  • Offline
  • Registered: 2009-03-19
  • Posts: 196

Re: robots.txt

Yes, the default keeps crawlers away from your apps. Robots would find the apps through the application directory and the featured apps page, even if you don't report them to search engines. There is currently no custom robots.txt setting, only allow or disallow.

3 Reply by Urdna

  • Urdna
  • Member
  • Offline
  • Registered: 2009-06-13
  • Posts: 20

Re: robots.txt

So how would I go about changing it for an app of mine?

Thanks,
Urdna

4 Reply by dvbportal

  • dvbportal
  • Member
  • Offline
  • Registered: 2009-03-19
  • Posts: 196

Re: robots.txt

It turns out, the setting is currently super admin only. Please name your app(s). I'll let it change for you.

The next platform update will make the setting available to all admins.

5 Reply by Urdna

  • Urdna
  • Member
  • Offline
  • Registered: 2009-06-13
  • Posts: 20

Re: robots.txt

They're called dhs, 180 and Andrew-Fasano.

Thanks so much smile

6 Reply by dvbportal

  • dvbportal
  • Member
  • Offline
  • Registered: 2009-03-19
  • Posts: 196

Re: robots.txt

Those apps are now followed by robots.