No, the Alexa Toolbar is not spyware.
Spyware is defined as hidden software that is installed on your computer without your knowledge. It generally collects your personal information for advertising purposes, and provides you with no value in return. If you would like a more complete definition of the term spyware, please seehttp://www.webopedia.com/TERM/s/spyware.html.
If you feel that the features of the Alexa Toolbar are not of value to you, the Toolbar is easily uninstalled using Add/Remove Programs in the Control Panel of Windows, or from the Add-Ons Manager in your web browser.