AJAX and web crawlers/advertising

I've been looking into the topic of how to both have an AJAX/DHTML website using a toolkit like prototype or an interface infrastructure like Google Web Toolkit. It boils down to the fact that if you need to let the web crawlers in you have to give them something non-DHTML/AJAX to consume. This can be done a few ways it seems, including intercepting page requests and directing to different handlers based on who's requesting and having a parallel site, one AJAX and the other plain old HTML. Another option is to limit the AJAX to page elements that make things more convenient and functional for the user; things like hide/show login areas, etc. Again, the point is to have a strategy to let the web crawlers in.

I'm thinking about all of this in the context of setting up a site that needs to be indexed properly by Google in order for Adsense to work properly. I'm leaning toward the route of having an entry point for web crawlers and another one for the application with exactly the same core content visible on both. The web crawler content would be optimized to provide maximum meta data and minimum extraneous bulk. I'm leaning to this solution because I am intruiged by the Google Web Toolkit and its ability to be used to build a content-rich site. Of course, I'd have to get a half-decent development machine to do this as well since the GWT uses a model whereby the application moves from a Java one in development to a JavaScript on at deploy time. The Java/Eclipse/etc. part is pretty resource intensive methinks.

Two bug firsts

Two of my bugs have made their way on bugguide.net as firsts in their category. One is a picture wing fly. I found it on the stump of a recently cut-down tree in our drive.


The other is a sap feeding beetle. This one was on a moon flower in our neighbour's small but wild front yard.

Ladybug takes flight

I've not taken photos of bugs since May, but while camping at Selkirk Provincial Park I got this fluke shot of a ladybug taking flight from my thumb.

A Java vs. Ruby example

Here’s some Java code I borrowed and wrote to extract multi-word tags surrounded by quotes from a string and add them to a list

// Now extract all multi-word keywords delimited by spaces
// but not surrounded by quotes
p = Pattern.compile("\"(.*?)\"\\s*");
m = p.matcher(keywordString);
sb = new StringBuffer();

while (m.find()) {
 // Get previous match and add it to the keywords list
 String kw = m.group();
 if (! "".equals(kw)) {
   kw = kw.trim().replaceAll("\"", "");
   keywords.add(kw);
 }
 // remove the current match from the string
 // and thus from consideration
 m.appendReplacement(sb, "");
}
m.appendTail(sb);
keywordString = sb.toString();

Here’s the Ruby equivalent

keywordString.gsub!(/\\"(.*?)\"\s*\/) {keywords << $1.strip}

It’s not the lines of code (although the Ruby code is 1/3 the size) that I notice so much as the unintuitive Java API. appendReplacement? What’s that? Verbosity does not add clarity to the Java and the lack thereof does not detract from clarity for Ruby.

OpenID

I've been investigating (and testing) OpenID lately. The cause, by the way, would be helped if openid.org's url worked without the www subdomain. Openid is an api that allows a person to claim ownership of a url and through that to claim an identity of sorts. It's not a way to prove that a person controlling a url has a certain name, so it's not an authentication mechanism. What it allows is for identity verification to happen in one place rather than over and over and over again in multiple places. An OpenID url I've gotten is ian.marsman.myopenid.com. With this I can log in to livejournal.com, zoomr.com, and other OpenID-using sites. The benefits for the user include the need to have a single identity verification location that can be used on multiple sites. A web application developer using openid as a login mechanism doesn't need to worry about account registration, which is rather nice.

The business model for providing and managing OpenID accounts does not seem to be that promising if that's all one is providing. The API is public and client and server libraries are available in a number of programming languages. One would need to use account management as a way to gain credibility for an identity management consulting business or add extra services on top of the base account management. claimid.com is doing this (or will be once they're out of beta). They seem to want to offer a way for people to point to various urls about the 'net and say "this is mine or about me". They also offer the ability to register other OpenID urls with their site which can be verified by them (the OpenID api allows for this).

In any case, I've installed and gotten running the Ruby version of OpenID. It's available as a gem, which I can't install easily on my non-root-access account. I've thus put all openid libraries under the lib directory of my rails application. This works pretty well. The sample openid_login generator is found and thus can be installed if one puts it in one's ~/.rails/ directory. openid_login.

One gold rush identity management system I'm not crazy about is i-name. i-names can look like "=ian.marsnan" for a personal i-name or "=@myorg*ian.marsman" for a person at an organization. I'm not crazy about this setup because the going rate to register an iname is twenty bucks US. For this, one gets more control over who you give what personal info to. However, OpenID has the ability to create profiles and choose which profile to give to a site that's requesting permission to access one's identity. i-name is an api designed by rather large organizations. OpenID is more grass roots, although Verisign is on the standards committee. Who knows how things will pan out. Both offer the hope of single sign-on. i-name seems more targetted at uses for businesses like corporate identity management and online banking signon authentication. It's a big topic and I'm starting to wander. At the moment all I want is a way to offload user signup management and give people a way to avoid adding my site as another to keep membership track of.

DIGITAL MAOISM: The Hazards of the New Online Collectivism

A great piece on the risks involved in building knowledge using groups. Essentially, the author seems to be suggesting that the results of consensus is not necessarily genius or deep insight, but rather blandness or at the very least, something lacking in boldness and insight. A great read.

The beauty of the Internet is that it connects people. The value is in the other people. If we start to believe the Internet itself is an entity that has something to say, we're devaluing those people and making ourselves into idiots.

The trick is to allow people to have a say and interact while preserving the individual. The article's not a slam against algorithms or Wikipedia, but rather a call to carefully consider how peoples' individual and collective wisdoms can best be used without wiping each other out.

Potential calendar problem in Java!

Java has a GregorianCalendar class (representing a data's year, month, day, etc.) with a get method that takes an integer argument and returns things like that instance's year as an int value. This means that the value returned will be invalid for year values beyond 2,147,483,647 (the maximum value of a Java int)! I think that's the time in Babylon 5 where humans escaped their physical bodies and left our solar system to avoid the impending explosion of the sun. Seriously, of more concern to me is the pain in the neck date and number parsing and formatting is in Java. Arggh! Things like this bug me:

int remainder = new Double(
Math.IEEEremainder(
  new Double(i).doubleValue(),
  new Double(startMonth).doubleValue())/12
).intValue();

What about Ruby's

 (27 % 12).to_i

If that's syntactic sugar I'll risk the cavities.

Letter to the Discovery Channel

You have an informative article on newly discovered species of assassin spiders from Madagascar. Unfortunately, the person writing the story chose to depict them as bizarre and ugly, with phrases such as "recognized by their peculiarly ugly, stretched-out necks and sword-like fangs" and "venom-loaded fangs, attached to the ends of grotesquely stretched-out jaws". Perhaps the author has a particular dislike of spiders or perhaps he wanted to avoid sounding too "sciency". Whatever the reason, I think that descriptions of well adapted spiders as ugly and bizarre does both the spiders and the intelligence and curiosity of your readers a disservice. Most of those who read the article will most likely be there because they find spiders interesting, not horrific. Besides, from the first image the spiders' jaws, head, and necks look a lot like the beak, head and neck of a pelican or stork. One doesn't hear of the blue heron as "having a grotesquely long beak adapted for stabbing its unsuspecting prey from above with lightning speed". In parting I'll leave you with a link to some images of beautiful, well adapted spiders

Chandler is coming along

The Chandler project, started by Mitch Kapor of Lotus fame, is finally starting to look polished and usable. At the moment, the most polished component is the calendar, which has some beginning ineroperability with the server using CalDav. Previously, there were no screenshots of the application, mostly, I suspect, because it was so ugly. It takes a long time to get the backend for such a data-driven application going and yet more time to get the backend hooked into the front-end. Congrats!

The recent White House wiretap affair and technology

ars technica has a great overview of the probable technology behind the recent non-authorized wiretapping affair. I am not at all a fan of a leader, American, Canadian, or other, feeling that he or she has the authority to eavesdrop on the lives of its citizens with not even minimal oversight. However, that is not what I am most interested in here. The ars technica article outlines an eavesdropping system that essentially samples up to one percent of phone conversations being made at any given time. Knowledge of details such as the country being called is used to home in on the most desirous calls, but the fact remains that an automated system will still come up with a few thousand possible phone lines to monitor every day. The decision as to whether or not a call is of interest is made, at least in part, by looking for keywords in call conversations. So, what we end up with is a surveillance system set up in many ways as a high-tech fishing expedition. The premise for a judicially approved wiretapping system is that wiretaps will be made based on pre-existing evidence. This is at odds with a technology-driven system that uses keyword matching as a large part of its algorithm. No free and fair judicial system would approve wiretapping of its citizens purely on the basis of their having uttered forbidden words while talking on the phone, yet that is what this technology demands. We have a basic conflict here between a technology and its needs and a democracy and its underpinning values and rules. The technology's judgements become our new judgements. Now, instead of being guilty of having done something illegal, we are guilty of having uttered something forbidden. The axis of our system of justice and valuations of freedoms is moving to a new centre of gravity.

The ars technica article points out that for all its technological beauty, the system being pushed is not one likely to notice only criminals. It is very likely that a machine-driven system will end up flagging lots and lots of non-criminals as potential terrorists or drug dealers or whatever. Once flagged by a beaurocracy, especially one that loathes oversight and transparency, one is always suspect. We seem to be swimming farther and farther into dangerous waters with very little thought to the implications of our actions. The love of technology instead of thoughtful human insight is not only an American problem. Perhaps they are the early adopters. Perhaps Canada's spies are even less accountable than ours. Perhaps the US system is more leaky or more transparent. The point is that as long as we undervalue human judgement and insight and the checks and balances that keep us free from tyrrany and overvalue technology, we are at risk.

Update: Josh Marshall comments on this as well

From a technological point of view there's not really much outlandish about this at all. This is just the sort of thing the NSA is in the business of doing overseas. But you can see how this would just be a non-starter for getting a warrant. It is the definition of a fishing expedition.

May I submit that wholesale eavesdropping on people around the world is outlandish, especially if it's done with minimal oversight. The bigger the machine the bigger its appetite and the more likely it will turn on its creators.

Testing Flock Blog editor

Well. Here goes. Downloaded the preview release of the Flock browser and am testing it out. Uses de.licio.us to store bookmarks and hooks into blogging. We'll see how revolutionary it is.

Be-pollened Eastern Carpenter Bee

Latin name: Xylocopa virginica

Bees are busy and difficult to photograph. I did however manage to get get this side profile of an Eastern Carpenter Bee covered with a remarkable amount of pollen.

Phidippus clarus female hiding out on Parsley leaf

Latin name: Phidippus clarus

This little female Phidippus clarus was hiding out in our Parsley. The pose shown here is the typical defensive posture; crouching and ready to strike while maintaining maximal ability to view surroundings.

Green Bottle Fly on Sweet Pea flower

Latin name: Phaenicia sericata

This is the unpleasant part of flies (besides their oftentimes diet of rotting flesh or garbage). The front of their heads looks like the front of a human skull, or at least the nose part.

Jumping spider - Sitticus sp.

Latin name: Sitticus sp.

This teeny Jumping Spider was found by Janneke on the sill of our living room picture window. It was only about 2mm in length, total. I brought it outside to our porch and took some pictures of it. It's amazing that something so small can be so complete and capable.

Red-banded Leafhopper

Latin name: Graphocephala coccinea

These bugs are cute and colourful! I don't know how it's adaptive to be red and blue on a green leaf. Perhaps it's less conspicuous in the spectral range visible by its predators. In any case, they look great to humans. Teeny too (about 7mm).

Green Bottle Fly on paving stone

Latin name: Phaenicia sericata

I am not normally a fan of flies, but yet I keep on taking pictures of Green Bottle Flies. It's the irridescence. Looking at them face-on is not pretty, I think because they have indentations similar to those one might see on a human skull. Of course, the fact that they start out as maggots in rotting flesh can be off-putting. In any case, horrid or not, this fly was grooming himself on the lovely walkway through our garden when I took its photo.

Since she's gotten new grandkids my mother simply refuses to use cool shots like this as her computer wallpaper. Imagine!

Long-jawed Orb Weaver on reeds

Old


New

Latin name: Tetragnatha sp.

I re-edited a shot of a female Long-jawed Orb weaver on some bent reeds to bring out the colour more (I blogged about it before). Now the colour is warmer, reminding me of how things look on moist, overcast days. What a spider!

Platycryptus on windshield

Old (non colour adjusted)


New (colour adjusted)

Latin name: Platycryptus sp.

Last night I used the GIMP to do more clean up an image I'd already posted to Flickr. The challenge was to reduce the visual influence of the blueness of the windshield on which the spider stands. The spider, thankfully, did not have any natural blue colouration, so turning down blue and magenta saturation accomplished this goal quite well. This allowed the subject to become much more visible relative to its background. Secondarily, I found that the Gimp allows one to define white and black reference colours in its colour spectrum selector tool. Doing this had the effect of removing a blue cast on the spider's white hair that I hadn't noticed earlier. It seems the camera's electronics where overwhelmed by the blue background and added the cast. In addition to removing the cast, the black/white baseline choosing made the whole picture more clearly defined.

I'm glad I've discovered the black/white baseline setting capabilities of the GIMP. It should help make my images much more attractive. I'm sure Photoshop does this very very well too. However, it costs $500.

Male Black and Yellow Argiope

Latin name: Argiope aurantia

This male Black and Yellow Argiope was literally hanging around above a female in her web, waiting patiently to mate. The female is visible here as the earth is visible to an orbiting satellite. One can see the male transferring some sperm to his palps. I read that males of this species die at the end of mating to inhibit other males from getting at the female. After resting for about twenty minutes the female will pick the male off and eat him, but but that time it's too late for other males.

Wolf Spider lateral view

Latin name: Hogna frondicola

This Wolf Spider was wandering about on the large stone I used as a background for the grasshopper photos I took. As with the grasshopper, I used a polarizing filter on shots of this spider. This spider had a total length of about 9mm.

Two-Striped Grasshopper frontal view

Latin name: Melanoplus bivittatus

Well, I don't get too excited about grasshoppers, but Janneke suggested I photograph one she'd caught and was keeping in her bug collection cage. She said that after release it would stick around for a while. True enough, when we placed it on a large hunk of stone it stuck around long enough to take about a dozen photos. They turned out ver well indeed. I used a polarizing filter to cut down on glare from the rock and reflection from the grasshopper.

Fishing spider on stone

Latin name: Dolomedes tenebrosus

This is a good rear view of a female Dark Fishing Spider (Dolomedes tenebrosus). She has a missing leg. This is not as uncommon as I'd have thought before taking so many spider pictures this summer. Perhaps they lose limbs in their nursery web or in territorial combat with others of their species.

Black and Yellow Lichen Moth on flowers

Latin name: Lycomorpha pholus

Also known as the Black and Orange Lichen Moth and the Black and Red Lichen Moth. Its colour varies by region. This lovely specimen was feeding on flowers in a garden by a restored grist mill we were visiting last Sunday.

Butterfly on Goldenrod

Latin name: Phyciodes sp.

Got just one shot of this butterfly on Goldenrod, but it was a good one.

Restored late 1800s mill

Near Brock University and the site of the War of 1812 Battle of Beaver Dams is a fully restored grist mill with accompanying mill pond. Beyond the mill is a spectacular waterfall and lovely river gorge. All of this is home to delightful insect life, including huge Millipedes and gargantuan Fishing Spiders. The skill and devotion put into restoring what once was a dilapited wreck of a mill is inspiring.

"The acquisition of Canada this year, as far as the neighborhood of Quebec, will be a mere matter of marching, and will give us experience for the attack of Halifax the next, and the final expulsion of England from the American continent."

-- Thomas Jefferson in a letter to W. Duane on Aug 8, 1812

I'm thankful for more peacable times.

Funnel Spider and prey

(1)

(2)

(3)

(4)

Latin name: Agelenopsis sp.

This past Saturday morning I put an ant in a Funnel Spider's web and photographed the ensuing subduing. The results were pretty graphic and kind of sobering. Number 3, shown here, I found especially troubling. One steps on ants, one poisons ants, one watches ants teeming about the garden. One does not always see an ant one has put in a web gaping as he's fatally bitten by a spider.

Long-jawed Orb Weaver on reeds

Latin name: Tetragnatha sp.

Long-jawed Orb weavers are neat, but not photogenic. They are too good at camouflaging themselves and don't look enough like regular spiders to stand out in a photo. Thus, when I post one on Flickr I don't expect lots of views, even if it's a photo I think is very good. Bright green grasshoppers are popular though and bright orange Colorado Potato Beetle larvae.

Jagged Ambush bugs on Goldenrod

Latin name: Phymata pennsylvanica

Finally! Got some shots of Ambush Bugs. They were on my list for this summer bigtime. They act a whole lot like Mantids, attacking swiftly from a position of ambush. They have a sucking mouthpart that they stab into their victims and use to suck out their body fluids. Notice the hooked ends on their forelimbs.

Fishing spider frontal view

Latin name: Dolomedes tenebrosus

Nancy found this huge spider on a stone in a creek where we were exploring. The creek flowed through a protected river gorge, providing a great environment for rather large insects to thrive. This is a female Dark Fishing Spider, one of three species of Fishing Spider I've come across in Ontario. It lives in North America east of the Rockies.

Although this individual was large (about 5cm), Nancy soon found an even larger one (about 7cm). I had not expected such large spiders to be found in Ontario outside of a zoo.

Note the missing leg on this spider's left side.