Others doing this kind of work: <http://www.zooknic.com>,
<http://www.registrarstats.com>, <http://www.sotd.info>,
<http://www.dailychanges.com>. But I don't think these folks use Whois
data, for obvious reasons (e.g. they can't get it).
The rise of GIF and other restricted access to HTTP Whois, when combined
with disabling of port 43 Whois, essentially puts an end to researchers'
ability to do the kind of work posted to
<http://cyber.law.harvard.edu/tlds/001/#topreg>,
<http://cyber.law.harvard.edu/people/edelman/name-restrictions/>, and
<http://cyber.law.harvard.edu/people/edelman/dotus>. Indeed, since HTTP
Whois is already pretty well locked down and since some registrars have
already disabled port 43 (contrary to policy, to be sure), this research has
already become sufficiently impractical that I generally don't spend the
time trying at this point.
I agree that an end to machine-readable on-demand data favors those with
bulk licenses of Whois data, e.g. T&T and MarkMonitor, as well as those who
can afford to hire these companies. This preference comes at the expense of
independent data analysis folks who surely don't have $10k per registrar per
year. My intuition is that the result will be higher prices from T&T and
MM, probably along with reduced innovation. Certainly there will be fewer
independent reports as to the sorts of domain name phenomenon that can only
be discovered using Whois data.
Some possible policy responses:
1) Require that port 43 Whois remain available to bona fide researchers.
Policy could require that researchers demonstrate an affiliation with an
accredited university or organization -- or not, depending on desired
treatment of independent bona fide researchers. (I would think most, but
not all, will hold some institutional affiliation.) Access could be
provided on a by-IP basis or on a passworded basis. This is probably not
unduly burdensome on registrars because I gather registrars will continue
operating port 43 Whois servers for their own internal purposes (authorizing
transfers, etc.).
2) Require that bulk Whois data be made available to bona fide researchers
on preferential terms (i.e. considerably less than $10k/registrar-year max
currently specified in registrar contracts with ICANN). Given the sensitive
nature of this data, it would still be important to make recipients sign
contracts as to use and distribution of the data. But to facilitate
meaningful research, the contract shouldn't prohibit partial republication
of the sort linked above.
3) Require that bulk Whois data and/or port 43 Whois data be made available
to some notion of "startup" Whois searching services that are unable to pay
the higher fees. Fees could be a function of these companies' initial
revenues, perhaps, or fees could be indexed to registrars' actual marginal
costs of distributing the data to licensees (e.g. bytes downloaded from FTP
server).