XModus eCommerce Blog

Network Solutions Ecommerce credit card theft could happen to your e-commerce system too…

2009-08-05T01:20:00.000-07:00

Two weeks ago Network Solutions Ecommerce the former MonsterCommerce folks) announced that approximately 573,928 customer accounts and their credit cards had been compromised -- but they were quick to point out that they had followed proper PCI DSS compliance procedures for storing the card data which is possibly how they were able to detect the breach in the first place. So, what they are saying is that they did everything correctly and still someone was able to steal a lot of credit card data. What appears to have happened, based on the reported details, is that someone was able to access the Network Solution (NetSol) servers and intercept the credit card details on the server itself – in other words the bad guys broke into their servers, changed the code and stole the information – let me explain.

Credit Card Data Vulnerability

When you visit a retail website and input your credit card information here are the main points where your personal information is vulnerable:

Over your shoulder: You first shoulder check to ensure that no other person can see you input your precious credit card and even more precious verification number (that number on the back of your card near your signature that tells the credit card company that the person inputting this number is holding the physical card). Being the clever one you are you always hunch over and block anyone from seeing you key in those precious numbers with your free hand -- so far so good.

From your computer: You have scanned and secured the computer you are working on to ensure that no one is recording every key that you press on your computer so you are certain that you have not allowed someone else to capture your unencrypted credit card information before it even leaves your computer… they can do this via the power cable you are connected to, if they are close enough they can monitor the signature from your keyboard or they have tricked you into installing a key-logger on your computer like this one for the Mac or this one for the PC (and these are the legit ones)... but again, your computer is clean and has anti-virus and anti-malware stuff installed (oh, and a firewall too) – so we've dodged another bullet.

From your web browser: You are certain that the web browser we are entering your information into is itself secure and that it has secured the channel between your computer and the retailer's web server using a valid and verified SSL certificate (the little yellow lock) and you are certain that the web server on the other end is actually who they say they are and not someone else who has taken control of the retailer's website. If you're scratching your head at this one let me simplify it… you have the latest version of your browser software and you read any security warnings or pop-ups from your browser (typically there are none but when they do appear you did read them right?) – again, you're doing all the right things so far so you’re confident that you're following proper credit card security procedures...

From the "website": You are certain that the web page that we are entering your information into is actually the retailer's server and not another site that has somehow tricked you into visiting their bad-guy website while masquerading as your target website. See #3 above and make sure you read the URL in your browser’s address bar, you know that long string at the top of your browser that should read something like: https://www.retailersite.com/something... and not something like http://www.retailersite.badguysite.com/something... but you are not worried here, you’re confident that the site you are visiting is your target site...

Okay, so you've successfully made it over the moat and to the castle's door to deliver your precious credit card so they will ship you the items you desire, now comes the twist.

From the web server: You are certain that no one has compromised the retailer's server and injected their own software between you and the retailer's e-commerce software – this of course is a trick question because this isn't your job and even I can't verify this one (even I have my limits). It's the retailer's and their service provider's responsibility and according to the news reports this is where NetSol appears to have fallen victim.

PCI DSS compliance requires that the retailer ensure that their physical security and software security are covered but this is where the story gets a bit vague (PCI folks please hold on, we'll get to this later in the program). What I mean by the story being vague isn't that the PCI DSS standard doesn't cover this, it does, it's that most of the claims that I see amongst my peers is that they have ensured that the data they store to disk is encrypted... good, because that is required. But PCI DSS is beyond just the software, it's also about the safety and security of the servers, their operating system and the application source code and this is where NetSol got hit, again according to the published reports -- the source code for their application was compromised somehow.

For the non-programmers in the room let's try an analogy (if you haven't left the room already). Suppose you visit your favorite restaurant and you give the waiter your credit card and he happily takes it to the point-of-sale (POS) terminal and swipes the card… no, the waiter is not the villain in this story (not this time), instead the guy who came the day before to "fix" the POS terminal swapped out the credit card reader with one that he made at home that reads the credit card information off of your card and records it to a little device inside the POS terminal that isn't supposed to be there. The device then takes the credit card information and passes it to the device which then encrypts it and sends it to the bank, etc. The problem isn't that the data wasn't encrypted it's that the data was intercepted before it was able to be encrypted and so your information has been compromised – again not by the waiter but by someone who planted a shim of some sort between where your information is input and the bank.

Now, let's look for someone to blame:

Did the restaurant owner verify that the guy from the POS-company installed a fully compliant device? Of course not, he's not technical he just runs the restaurant. In our story the retailers assumed, rightfully-so that NetSol would take care of this (and it appears they did in the end).

Did the POS-company knowingly plant this device? Unlikely, it's more likely that someone inside the POS-company did this or someone else knows enough about the device to swap it with their own device… one credit card terminal looks like another if someone has the time and access to swap the devices. In NetSol's case it was either an inside job including a former contractor or former employee, though this is unlikely, but possible, or someone externally gained access to the server or software somehow and compromised it (most likely).

Did anyone at the restaurant verify that the guy fixing the POS-terminal was actually from the POS-company? Probably not, it's called social engineering and if someone shows up wearing the right clothes and the necessary name tag/ID the support staff aren't going to verify anything, they got stuff to do. Ah... this doesn’t apply to the NetSol story unless someone claiming to be NetSol tricked all of these retailers and from the sounds of it that's not the case... but it lets me round out my list with three things and it's a security issue nonetheless.

Let's get back to NetSol for a second and explain what they (and your current provider) should be doing:

Source Code Control: No one should be able to modify their source code without them knowing about it. Now, I don't know their policies around letting customers edit the source code but assuming that their customer’s are not permitted or able to edit the source code then this was either an inside job at NetSol or someone got to the servers, which leads me to #2.

Server Access Control: No unauthorized persons should be able to access the server's such that they can get to where the application is installed and change it. This is like physical security, i.e. you cannot swap something out if you can't get to it. This is covered by PCI DSS (see PCI folks I said I would cover this later) and it's just plain-old common sense network administration – only give the minimum of access and log and audit all of the time.

Restrict Third-Parties: There is a third possibility namely that someone wrote a plug-in for the NetSol application that was popular and this plug-in somehow injected itself into the unencrypted stream of information or was able to access the data in an unencrypted format before they were able to encrypt it. This should not have been possible but since I don't know their application well enough to comment on it I'll just leave it as a possibility situation albeit unlikely in this case.

Okay doctor, what do I do now?

So, you have your own store and you're sweating right now because you don't know if someone has done this to your application and you're asking yourself is there anything that a non-technical person can do to prevent this? What questions should I be asking my e-commerce software/service provider right now to ensure that I don't fall victim to this – oh, and if you are thinking of switching away from NetSol and I've been seeing a lot of companies advertising their software on the back of poor NetSol you should be asking them these tough questions too before you jump ship:

Trustworthy Employees: What process do you have in place to ensure that the people who work on your source code have not planted a back-door or their own logging code into the application such that if they get tired or fired they just flip a switch and start skimming card or customer information? The right answers include employee contracts, background checks and code audits.

Source Controls and Audits: What process do you have to ensure that someone hasn't compromised the software on the servers? The right answers include verifiable* code audits on all of the installed versions of their application to ensure that no-one outside of the authorized personnel have been able to update the software on your servers and the worst-case scenario is that there is a complete audit trail... oh, and saying that they use source control is like saying that you have a sign-in sheet at the front door to your office and that you are certain that the bad guys will sign their real name on the sheet as they pass by to do their dirty work – it ain’t gonna happen.

Server Security: What process do you have to ensure that someone hasn't compromised the server itself? The right answer includes a combination of tools like Tripwire, anti-virus, root-kit scanning and standard system-administrator tasks that monitor and audit servers. Again, you need to verify that they actually do these things and not that they just say they do.

Disaster Recovery Procedures**: What procedures do you follow when something bad like this happens? The right answer includes at a minimum an action plan related to securing the information and servers and working with you to meet disclosure requirements to your customers, your bank, etc. This is something that NetSol has reportedly been working to do with their merchants and their merchant's customers.

* When I say verifiable I mean that they actually have a documented policy that they will show you (under NDA) that spells out their audit and review process and that they guarantee that they will conduct these at regular intervals to ensure your safety and security.

** Okay now I've offended all of the Business Resumption Planning (BRP) people by using their term "Disaster Recovery" but for our purposes this is when something bad happens and that's what BRP is for and so I'm using it... my blog, my terms.

Did I scare you?

I hope I scared you just enough to realize the vulnerability that you may face each day with your software and not so much that you have shut down your online store out of fear of bad things happening. I happen to have been a system administrator and a programmer and I know that each group can get locked into thinking only about their individual piece of the system thus ignoring the system as a whole and leaving a hole just big enough for someone to take advantage of as it appears was the case with NetSol. I also know that with proper procedures and policies this can be avoided or at a minimum it can be detected and stopped with minimal damage to your company and its online reputation – as appears to have also been the case with NetSol in that they did catch it. Put yourself in the shoes of the retailers who must now report to their customers that their service provider, albeit a very big one, allowed their information to be compromised… would you as a consumer be placing the blame on NetSol or on the retailer?

What is NetSol doing about it?

To NetSol's credit they have launched a website to answer questions for their merchants and their customers including letters and such that will go out to customers and they are offering free credit monitoring for all potentially impacted customers through a third-party agency.
Their incident website is here: Ecommerce Security: What Happened
There are still individual US State laws that must be followed and if you are interested follow the link.

Any questions?

I hope I have given you enough to take action immediately but feel free to ask questions in the comments below or contact me with your questions directly:
Chris Kerslake, President, XModus
Email: chris@xmodus.com
Twitter: chris_kerslake
Phone: (604) 732-7337 x101

Sources for this article:

[1] Network Solutions Hack Compromises 573,000 Credit/Debit Accounts
[2] About the PCI Data Security Standard (PCI DSS)
[3] Ecommerce Security: What Happened (Network Solutions' website to explain the situation)
[4] Mac Key-Logger
[5] Windows Key-Logger
[6] Configuration Control - Tripwire
[7] Business Resumption Planning: Justification, Implementation & Testing
[8] State Security Breach Notification Laws (As of July 27, 2009)

Privacy Filters Blocking Google Analytics eCommerce Tracking

2009-06-22T10:02:00.000-07:00

Some of our customers use Google Analytics' eCommerce tracking on their website to record order revenue information into Analytics for matching with their Google AdWords spend. In theory this should be perfect because they can record their website analytics and at the same time also record their marketing spend on Google. Unfortunately Google Analytics relies on a script embedded within the order confirmation page to be executed at the time of purchase and this particular method of reporting is falling victim to various privacy protection mechanisms employed by online shoppers today.

Here is what happens...
Shopper places an order on your website and during the order completion phase a Google-supplied script is embedded into the order comfirmation page. The shopper's browser is then asked, implicitly, to execute this Google script to send the data to Google. This script takes some of the order information and transmits this to Google for reporting within Google Analytics. Unfortunately this relies on the shopper's browser to cooperate with the implicit request and send the information to Google, which the user's browser can choose not to.

What we discovered...
What we discovered is that on average approximately five percent of transactions were not being sent to Google. We checked and noticed a pattern of newer browsers specifically FireFix, IE, Safari, and Sunbird were showing up in our browser logs for the 'missing' orders. A bit of poking around led us to a variety of discussions about these various browsers and how they can thwart various tracking techniques. We did a couple of quick tests and confirmed that in fact the two that we tested, IE 8 using InPrivate privacy mode and FireFox 3.0 with AdBlock Plus could block Google Analytics (among other tracking scripts and pixels).

Is it only Google Analytics that is blocked?
No. The basic assumption that all tracking pixels, cookies and scripts rely on is that the user's browser will act in a consistent manner and transmit the necessary and requested data from the browser back to the third-party's server. If this trend continues and browsers start to block more and more of these scripts, pixels and cookies then the reliability of these methods of tracking will decrease as well.

Are there alternatives?
Yes, there are two classic ways around this. The web server that is being called already has all of the information that Google Analytics is looking for and thus could submit that information directly to Google, thus bypassing the user's browser, or it could choose to process the information itself. The advantage of having the user's browser do it is that the user's browser controls the privacy level the user wants to enforce and the web server does not have to do any additional work. If the web server must process the request itself then it will require additional resources to do so something that is avoided today by having the user's browser do the work for it.

Is five percent that big of a deal?
That depends. Obviously knowing that 5% of sessions are currently being impacted allows for factoring in this level of error and using simple mathematics our customers can extrapolate the difference. Where this becomes problematic though is if you are using these numbers and assuming they are accurate and thus could be making inaccurate (or even incorrect) assumptions. Also, it's only five percent today but as users upgrade their browsers and add these privacy mechanism in then we expect this number will increase as well.

How did we know there was a difference?
Our Rebus eCommerce software receives all of the orders and processes them so it has the actual order totals, revenue, costs, etc. and it is Rebus that transmits the order details to Google Analytics. One of our customers was using the Google Analytics data for a report and was comparing the Rebus report and the Google Analytics report and contacted us to find out why there was a difference. They pulled the raw order numbers and discovered that a a bunch of orders were missing and reported it to us. We checked our logs and could see that we were sending the information to Google and so we looked further to see if we could discover the reason and that led to this article.

More information:
Google Analytics - How do I track e-commerce transactions?

AdBlock Plus Forum: how block Google Analytics

IE8 and Privacy

IE 8 feature thwarts targeted ads, Google

If you have a question about this topic or article or any other articles on this blog please contact me, chris@xmodus.com or you can call me (604) 732-7337 x101 or follow me on Twitter chris_kerslake.

Keep your customer's passwords safe

2009-04-21T15:42:00.000-07:00

A few years ago a friend of mine had the unfortunate task of being forced to fire an employee for cause. In this case the employee in question did something to my friend's online customers which at the time if his customer's had discovered it could have meant the end of his online business. The crime here was the result of a bored customer service representative (CSR) who accessed customer's email accounts using the same email address and site password the customer had used on my friend's e-commerce web site. This CSR didn't set out to break into customer's email accounts instead one fateful telephone call from a customer tipped this CSR to the problem of customer's using the same passwords in more than one place.

Here's how it happened...
One day a customer called in to customer service with a question about their order and during the conversation the CSR asked the customer to confirm some security information about their account, i.e. zip code, street address and account name. While finishing off the customer's inquiry the customer mentioned that he needed to also update his account password but could not accomplish this via the shopping site and wondered if the CSR could do it for him. Of course he could and so the customer read off the new password and the CSR dutifully entered it into their back end system, problem solved. The customer then made a problematic statement to the CSR to the effect that he was updating his password on all of his accounts, including this one. The CSR didn't realize it until later that the customer was employing a single password for all of his online accounts and had just given this CSR that password to all of his accounts! At first the CSR didn't believe the customer but didn't pay it any heed until later that night (he was working the night shift) he was bored and for some reason decided to test the customer's single password comment himself. He noticed that the customer had a Hotmail email address and so he visited the Hotmail website, entered the customer's email account and password and voila he was granted access to the customer's email account. He was shocked but also a bit excited as he could read this customer's emails. He should have stopped there but he didn't and if he had stopped there he probably wouldn't have lost his job several months later.

As with many addictions the CSR wondered if other customer's also employed this single password ethos and so he decided to experiment further. Unfortunately my friend's e-commerce back end did not limit access to customer's accounts and did not provide an audit trail or alerts to this type of fraudulent behavior and so during the normal course of the evening this CSR would record all of the Hotmail accounts he came across as well as their account password and then later when he was bored he would try and access their Hotmail accounts. As he slithered his way through these online customer's accounts he began to discover personal emails about things that he should never have had access to and I'm sure customer's would not have wanted anyone to see. Finding a few juicy nuggets led to a full-fledged addiction to snooping in other people's email accounts, an email voyeur of sorts. But like those before him he got too brazen and with his mind filled with other people's personal information he hinted to the wrong person one day and this led to rumors about someone snooping on customer's records... a misinterpretation of his transgressions but one that ultimately led to the IT folks putting a tap on all external access and discovering his exploits. Finally, suspecting that he was up to no good, and thinking it was online fraud, my friend contacted the authorities who monitored and recorded the actions for a possible criminal case. Luckily for the CSR his online voyeurism was only deemed embarrassing but not criminal (at the time) and so with transcripts of his late night exploits he was confronted and shown the door.

Moral of the story...
1. Don't use the same password on other sites, at least come up with a variant if you want to avoid this 'one password to rule them all' problem.
2. Don't allow you customer's passwords to be accessible by your staff.

Other variations of this act...
1. Changing the email account to one that the fraudster can access. They will change the email to their own, email themselves the password and then change the email back so no one knows.
2. Customer data exported for transport containing customer's passwords.

Solutions:
1. Secure customer passwords by not allowing anyone to see the plain-text password.
2. Allow passwords to be reset and emailed to the original email only.
3. Audit all password and email account resets.
4. Alert customers at their original email address if any of these events occur.
5. Review audit trails for patterns... most people won't stop at one.

How Rebus handles this
It's surprisingly a common question that we get asked by new customers, "How do we access customer's account passwords so we can change them.", to which we reply, "You can only access their password in plain text if you have the necessary permissions." They inevitably reply with either a "that's great", "that's too complicated", or "please don't allow us to do that". We also have an audit trail that allows customers to watch for this particular type of internal fraud (as well as others too).

Do you have a question or comment? Email me, follow me on Twitter (chris_kerslake) or give me a call on the phone (604) 732-7337 x101.

Why you should care about Twitter

2009-03-29T21:07:00.000-07:00

I've been using Twitter now for the past month and I have to say that I can appreciate why people are excited about it but I can also appreciate why not everyone is keen on it, bottom line: lots of people are doing it, it's still very early and it can take a lot of time.

What is Twitter?
Twitter is an online service where users submit short text messages, up to a maximum length of 140 characters at a time (called 'tweets'), to individuals or simply to the steady stream of other 'tweets'. Think of a web page where new information is appearing every time you refresh the page from individuals telling you about themselves, about information they found, asking questions and just talking all of the time. There is a now infamous article by Guy Kawasaki, one of the most prolific Twitter users and biggest cheerleaders for Twitter:

Late one night in a hotel, I discovered I hadn't brought a MacBook power supply, and I was leaving early the next morning for a remote location. I posted a message to Twitter, and within 10 minutes, five people offered to bring me a power supply; one delivered it to me within an hour.

I haven't had the same response as Guy and when I have asked a question (so far 3 questions) I have not received any responses other than a new group of 'followers' who triggered on some keyword in my Tweet. For example I asked a question about Customer Relationship Management (CRM) and within a few minutes I received a wave of new followers and one that sells Sage CRM... my question was about something related to CRM not about me wanting to buy a CRM.

What is the point?
Initially I was skeptical and unsure of why I would want to keep announcing my every single move to the Internet. I decided early on that I would tweet only interesting or relevant things, not that my son just went potty or that I'm out for dinner at a particular restaurant. So when I initially signed up I was flattered that immediately all sorts of 'people' were interested enough in my profile to start following me, little 'ol me, why how cool is that... but then the reality of the situation set in quickly. Most of these 'followers' were simply automatic software programs, Twitter agents, that saw my new account and told Twitter that they were interested in following me. Being a good Twitter citizen (Twitterzen?) I decided to follow them too and there was the catch. The next day they would drop their following of me but I would continue to follow them and hear their message every time they tweeted. So I quickly developed some following criteria and for those that I thought were not interesting I let them continue to follow me but I did not return the 'favor'. I suspect the reason that so many people immediately follow those that follow them is a variation on THOMAS (The Human Oxytocin Mediated Attachment System) and thus works in their favor. On Twitter so many people are just trying to get the most followers that it's like a cross between narcissism and a game of who collects the most.

Why did I join?
I joined for a couple of reasons:

1. I'm a geek -- I'm a technologist and like to experiment with new technology -- I'm typically an early adopter and this is something new and interesting.

2. My friends are doing it -- Many of my friends were doing it and they told me stories of meeting and talking (tweeting) with people they had heard about but had never had a chance to talk to before.

3. I like to network online -- As I began to research Twitter I began to find a lot of people that I know from LinkedIn and Facebook on Twitter as well as some target customers and industry insiders on Twitter... all simply a tweet away.

4. I see gold -- I'm not alone in seeing a potential paradigm shift for online retailers to listen to and target consumers directly with specific advertising. One of my staff discovered an online retailer on Twitter, followed them and immediately received a message from the with a free gift offer as a thank-you for following them... she was flattered.

But what about ROI?
So far, like my friends before me, I haven't really seen any specific ROI. My investment so far has been learning and trying Twitter and I must say that it is very easy to spend a lot of time reading tweets and following links to read articles and such that people have posted. Let me tell you some of the people that I have tweeted with or who I follow or have followed: a competitor, some of my customer's competitors, potential customers, people I've read about but never met, anyone who I follows me and looks interesting.

Why should you join Twitter?
A constant stream of keywords from people 24x7 on everything; including your target customers, your competitors and other people you want to follow or who want to follow what you have to say. This is not the Matrix, you don't have to watch a black screen with green letters scrolling down it, it's just a list of comments from all the folks you are following.

Why you should not join Twitter.
Don't expect that if you join Twitter that they will come and you will somehow automatically be rich. Twitter is a social tool and that means you need to contribute to the community (i.e. work), just like blogging... in fact Twitter is considered micro-blogging, 140 characters at a time. Don't be afraid though, it's not just for bloggers.

Some "interesting" Twitter stories in the news recently:
1. Guy Kawasaki's request for a power cord from a hotel room...
2. Juror causes mistrial proceedings by tweeting...
3. US Congressmen post their experience during the recent US Presidential inauguration...
4. A Silicon Valley executive tweets during a break-in at his house...
5. Actor Ashton Kutcher posts photos of his wife Demi Moore and then tweets about it...

Are you using Twitter? Let us know what you think about Twitter and tell us your stories about Twitter in the comments below... or email me with your story, or better yet, follow me on Twitter! chris_kerslake - http://twitter.com/chris_kerslake.

Go back to the floor to reconnect with your business and customers.

2009-03-29T17:03:00.000-07:00

Recently Jeff Bezos, CEO of Amazon.com decided to spend a week working at one of Amazon's distribution centers. From the article:

Jeff Bezos is spending this week working in an Amazon distribution center in Lexington, Kentucky (AMZN). He apparently wants to see what it's like to be a rank-and-file Amazon employee. More CEOs should try that once in a while.

Like many entrepreneurs a friend of mine started off in the family business, starting off in the stock room, and growing with the business. Recently one of his key employees needed an extended time off and rather than hiring someone into the position or promoting someone temporarily he decided to take the opportunity to return to his roots and so he took over the vacant customer service manager position, in essence returning to the floor.

Now, there is an argument that the CEO doesn't have time to stop doing their CEO job and to be fair, he took on both positions at the same time with a known end time. His decision to step into the customer service position was, in his mind, an opportunity to answer the phones and talk to (his) customers again, something he hadn't done in a few years and here is what he found:

1. Staff were spending too much time on the phone answering questions about when customer orders would ship. When the umpteenth customer asked the same question during a call he decided to ask the next few customers what their expectation was and he got different answers and so he looked into it and discovered that the wording on their website was causing many of these unnecessary customer contacts. They reviewed the wording on their order confirmation page and also their order confirmation emails and discovered that the wording was ambiguous and after a couple of wording changes these calls almost completely stopped.

2. Customers were calling in and emailing asking for help with tracking their orders. Odd he thought, the tracking number is on their emails... until he asked a customer why they didn't just use the tracking number on their shipment confirmation email and the customer said that there must be a problem with his e-commerce system because the tracking number was blank! As you can imagine an investigation ensued and it turned out that some USPS packages were being sent out without a tracking number because the shippers were trying to keep costs down and so for low value shipments they weren't choosing to use a tracking number (an extra $0.18 per shipments). What the shippers didn't realize was that their eighteen-cent saving was actually costing the company more in dealing with customer contacts. A change was made to their emails text and processing to tell customers without a tracking number that their packages would take a certain number of days, instead of being blank and again the call volume dropped substantially.

So, even if you aren't the CEO of a company, returning to the floor today is important and here is why you need to reconnect with your customers:

1. Your staff are doing things that are costing you money (and they don't know it). You will find things as you do their jobs that you could eliminate or streamline. You will find that they are doing what they think is what needs to be done but don't typically have the authority to make changes or the goal of keeping costs down. You however will see these opportunities quickly and you have the authority to make the changes necessary to fix them. To be fair, in many cases that I have seen the line staff do know that things should be done differently and they may have even told you about them but until you actually see them for yourself you may not give them the authority to make the necessary changes.

2. Talking with your customers will help you find new opportunities or products. You pay good money to attract customers to your web site and then when you lose them it's so disappointing and damaging to your company. Talking directly to your customers will once again expose you to their problems and as any entrepreneur knows, problems are opportunities. When you talk to your customers you will find your next big idea.

3. The real world doesn't work the way you expect it to. When you return to actually doing the job, whether that is packing a box, picking an order or answering a customer's phone call you also challenge your mind's internal model of how the business runs with the realities of how your business actually runs. This reality will either inspire you or depress you, depending on what you find, but it will also put you in a position to correct the problems with your knowledge and authority.

Have you returned to the floor yourself or been there when the CEO did so, tell us about it. It's worth noting that the comments on the Bezos article I mentioned above has some great related stories. I know from my own personal experience that challenging how things are done and doing them yourself from time to time is a great way to discover issues and fix them.

Don't Make Customers Test Your Website – It's Cheap and They Won't!

2009-03-02T23:31:00.000-08:00

When was the last time someone other than you and your team reviewed your website? Your customers visit (and test) your website every day but have you ever asked them to comment on your website – other than having them comment by just leaving your website frustrated because they can't find what they are looking for or you possibly seeing them exit, via a page bounce report, after you have already paid for the click to bring them there in the first place? Simple usability testing is a great way for you to review your shopping site without spending much money and before your customers do.

Joel Spolsky (www.joelonsoftware.com), founder of Fog Creek Software and software best-practices teacher, notes in his Joel Test that "hallway usability testing", grabbing average, nontechnical people at random from the hallway and having them test or review your work, is a simple but effective way to improve your software. Since our customers are not themselves software companies we work to educate them about the importance of software testing – we test our software before we give it to them but once they change the user interface they need to test it again for usability.

Steve Krug, author of, "Don't Make Me Think, A Common Sense Approach to Web Usability", my favorite book on the subject of website usability, has a couple of quick points that I think are important for testing – because most people don't like to do testing:

1. If you want a great website you have to test.
We've already established that you need to test before your customers do. Customers are the worst testers because they just assume that your site will work the way they expect it to and they are not forgiving if it doesn't. This means that if you paid money to get them to your site and so when they are unhappy you not only lose that direct marketing money but also any possible future revenue from them as a loyal returning customers, not to mention their network of friends – "Their site was buggy or hard to use", won't draw their friends to your site. Also, visitors are busy and rarely will any of them drop you a note of any kind to let you know that there is an issue with your site. We have seen sites with Customer Experience Management (CEM) links and buttons – the "Report a problem with our site" – but our own experience on our own website and discussions with others that have employed CEM themselves is that it doesn’t work as you expect it to work in catching issues because people are just too busy and CEM is not for testing for bugs, it's for providing another channel for customers to communicate with you.

2. Testing with a single tester is 100% better than testing with none.
The math is simple, if one person tests then you are 100% better off than having zero people test your site. A few caveats though, not everyone is going to be a good tester and testing is not a one-time event. Testing needs to be done every time you make a change to the way your site functions.

3. Testing one user early is better than testing 50 users near the end.
Testing at the beginning is cheaper and better because once you build something there is all the extra effort that you must expend that you wouldn’t have had to have expended if you had tested earlier. Imagine that you are building a house and you decide after the house that you want another bathroom up stairs. If your house is finished then you have to wreck the walls, floor and plumbing to add the extra bathroom but if you had tested earlier and discovered that an extra bathroom was needed you could have added the bathroom at the start and not had to suffer the cost of building once and then wrecking and rebuilding a second time.

How to test your website on the cheap:
Steve Krug calls this "Lost Our Lease Testing":

1. Who are these mythical and magical testers? A tester is any reasonably patient human being, ideally someone who uses the web and has some experience buying online. Try to find users who reflect your target audience but don't get too caught up on this point because remember that having someone is better than having no one. Also, if you get more than two testers you are more likely to notice the difference between folks who are good testers and those that are not – take it from me the difference is huge and finding a really good tester is critical and hard – you don't need the best (yet) you just need someone who is acceptable and will get you started.

2. How many testers? Ideally 3 or 4. You want more than one so you get at least two opinions and when you get three or four then you should start to see patterns of problems. A single person is still better than zero though and two is twice as many people... you get the point.

3. Where to test: In person at your office or conference room where you and your team can observe them in person and possibly even video tape them. From personal experience, avoid, where possible, having them do the testing remotely and phone it in. If this is all you can get then do it but ideally you want to see them in person because you will be amazed at the level of detail that you will catch just watching someone try and navigate what you thought was a completely intuitive website! My favorite moment is when they fill in the form "wrong" and it causes an issue with the application – what were they thinking is usually the comment, to which I respond, "fix it".

4. Budget: $50/person or lunch or dinner depending on whether these are your friends or not.

5. How long should they test? No more than 1 hour at a time – think of how long your average shopper on your site spends on each visit, one hour is a long time to get through all the basics and some of the more advanced stuff. Ideally you want them to perform at least the basic tasks of buying a product, searching for a product, browsing for a product, and checking out. If you have time get them to also try and cancel an order, check order status, and even find answers to their questions. I would suggest that you record all the different ways that these folks use your site so that each tester is given the same set of tasks and you can compare them. Ideally though you want to start by just telling them to buy a couple of products and tell them to do it the way they always shop and see what and how they do it – I guarantee you will be surprised.

The take-away:
Test your website because your customers won't and it's cheaper, faster and easier than letting your potential customers leave because of something you could have found and fixes for the price of some pizza and beverages.

Turning Away (Known) Bad Customers

2009-02-23T21:58:00.000-08:00

Credit card charge backs are like thefts to retailers and in this case the retailer got the last word... the second time.

I received a telephone call from a friend recently to tell me that she had just turned away a sale on the phone. The shopper that had called in had placed an order with her company before and while she was placing their new order she noted that this particular shopper had done a charge back on their last order. A charge back is when a credit card customer disputes a charge on their credit card with their bank and the bank then disputes the charge with the retailer and unless the retailer can provide real tangible proof that they shipped an order to the card owner then the retailer is charged a fee, sometimes as much as $75, they get a warning from their payment provider (too many and you lose your merchant account), they lose the sales revenue, the shipping revenue, and they also lose the goods and the shipping cost to the 'shopper'... so charge backs are bad and merchants take them very seriously (and personally). So, knowing that this particular person had previously done a charge back she stopped the call, asked the 'shopper' to hold for a moment and went and got the owner. The owner verified the shopper's information and then stopped the order and told the customer that he was not welcome to buy from them any more due to his previous charge back and he ended the call.

This was a case of, "fool me once, shame on you, fool me twice, shame on me". I suspect the caller was shocked. If the caller was a scammer then he might try again and next time their fraud rules should catch him (assuming he uses the same name, address or credit card) but if he doesn't then at least he knows that this friend of mine won't put up with it, nor should they.

For those of you who do not have a way of flagging problem customers you should at least keep a "watch list" of shoppers (I hate to call them customers) who have initiated a charge back against you so you can block them or challenge them. Obtaining the credit card's card verification number is an effective, but not fool-proof, way to protect yourself from this. You should also verify that the shipping address is an approved address for the card holder to avoid them disputing the delivery to someone else and you should, where possible, always obtain a signature. I know in one case the courier did not obtain a signature, even though it was requested by the merchant, and the merchant was able to pass the charge back cost on to the courier company.

One question I was asked when writing this is, "How did your friend know about this person's previous charge back?". Our software has two mechanisms for protecting merchants from this... I didn't ask which one helped in this scenario but one of them did and I was happy to hear (and share) her story.

Do you have an e-commerce charge back story or want to know more about preventing and blocking repeat charge backs? Email me, chris@xmodus.com.

Online Cons - Shipping Address Quick-Switch Fraud

2009-02-22T20:19:00.000-08:00

Online fraudsters are a clever bunch and they will take advantage of the fact that most people want to help other people, from Paul Zak's great article, "How to Run a Con":

The key to a con is not that you trust the conman, but that he shows he trusts you. Con men ply their trade by appearing fragile or needing help, by seeming vulnerable.

Today's article is about an online con that we were made aware of a few years ago, we call it the "Shipping Address Quick-Switch Fraud" and here is how it works:

1. The conman places an order on your website using a stolen credit card and they input the victim's actual name, actual billing and actual shipping address to ensure that when it is checked against the actual credit card that it will pass any verification checks.

2. Once the order has been accepted on the shopping site they then pick up the telephone or quickly dash off an email, apparently from the victim, to the target shopping site claiming that they need the order re-routed to another location, say college or at a conference or whatever the case may be and whatever suits the particular items they are buying from the target site -- and thus affecting a shipping address quick-switch after the order has been placed and just past where most fraud checking systems exist.

Note that they are typically impassioned about this need, in one case they claimed that their son had called and had just broken this product and needed one for school, which started tomorrow and that the caller was even willing to upgrade the shipping, even though they knew it would cost more, if only the target could expedite the order.

Since most fraud checks are done by the credit card processor at the time the order is placed and since they have used the victim's actual address and name then this trick will pass fraud checking most times. And from the con man's perspective, the worst case scenario is that you will ship the items to the victim -- who will call very surprised and alarmed. What we found in one case was that the telephone number the con men provided was their own (we suspect they were throw away phones) and the telephone number did not match the victim's area code or match to the card's address -- now this was the case in one instance but if they take this con far enough they may also have the victim's telephone number and if they are calling you then they don't need to change the phone to a non-matching value that just might be caught by a fraud checking system.

How to protect yourself
1. Institute a formal fraud checking process for all orders and incorporate checks and rules that match your business and your products.

2. Be wary of expedited orders, especially requests to upgrade after the order has been placed.

3. Be wary of address changes after orders have been placed and ideally fraud check these orders again.

4. Don't trust any external information provided by your shoppers -- it only takes a second to verify a telephone number matches an address or that a credit card and address match -- and if they do not then you need to challenge the shopper... some will take offense and others will be grateful... again con men are very clever and they will be ready for you to fight back and will either run away and hide or will become aggressive and try and intimidate or shame you.

5. Share this story with your staff and study up on other cons, the more information that is out there the more you can be aware and protect yourself.

6. Define an internal policy around this issue and train your staff on the policy and the reason for the policy so they can help defend against it too -- a single customer service representative (CSR) may not have the gumption or confidence to reject an order unless they feel empowered with a policy and training, and even better tell your CSR's that when in doubt get a second opinion.

Won't a shopper's CVV2/CVC2/CID number protect me?
Not in this case. The CVV2 (Visa), CVC2 (MasterCard) and CID (American Express) numbers are strictly as a counter-measure against fraud introduced by the credit card providers when a credit card is not physically present (card not present transactions). In both of the cases that we have seen of this con, the con men had all of the information from the victim, and we suspect even their card verification codes. Here is the problem, the target websites didn't ship to the victim's address, they shipped it somewhere else so they didn't use the verified information and so the credit card company did not protect the merchant.

So, is the moral of the story not to trust anyone on the Internet -- yes, but it's not that simple. In this case if they had verified the information and challenged the con men they might not have shipped these orders. Also, if they had had a strong anti-fraud procedure in place or at least been aware of this particular con they could have at least challenged the fraudsters and possibly not shipped the orders. It is possible to dial your trust meter too far and not trust anyone and then you will institute policies that indicate that you will only ship to matching billing and shipping information and this will work but it will also limit your ability to sell to those people who, like me for example, prefer to have items shipped to their work and who have not added their work address to their credit card's list of allowed addresses.

Do you have an online con story to share or questions about fraud checking procedures? Email me, chris@xmodus.com.

SEO Voodoo - Sitemap.xml Will Make Your Site #1 on Google

2009-02-19T21:48:00.000-08:00

It just keeps happening, the voodoo that is, it just keeps coming my way and I have to say that I understand that during these "tough economic times" everyone has to make a living and I understand that some of the folks who make cold calls for technical companies are not themselves technical but please, please, please at least tell the truth.

One of my friends received a call a few weeks ago from a company that started something like this:

Hello Mr. Kerslake, my name is Jimmy and I was on your website this morning and noticed that you don't appear to have an ex-em-el (XML) sitemap. Did you know that having an XML sitemap is critical to getting your website to the top of Google, Yahoo!'s and MSN's organic search results? And, since I noticed you don't have one on your site, or at least one that I could find, I wondered if you would be available, and interested, in talking with our chief search strategist, Louis?

My friend, being a salesman himself, declined the invitation to talk to Stan but did collect the contact information for Louis and Jimmy and immediately fired off an email to me in fear that somehow his great website was missing this critical and chart topping mystical thing called an XML sitemap. I laughed when I read the email and then replied back with a link to his website's XML sitemap and a link to the sitemaps.org website so he could see the following quote:

Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata. Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.

So, you may be asking yourself, what about my website, it doesn't have a sitemap.xml file so does that mean I'm losing money?! The short answer is probably not and the longer answer of course, is that it depends. First of all, let's start with what an XML sitemap actually is and why you may or may not need one -- though you should have one and it's trivial to create one.

Search engines like Google, Yahoo! and MSN use software called web crawlers (or bots) that collect and index web pages on your website every day. These crawlers are sophisticated pieces of software that analyze each page and look for links on your website, and other websites that link to your website, for ways to index your website. If you have links to all of your pages somewhere on your website then the crawlers will find and index your complete website. If however you have a lot of dynamic or temporary content or say you are the New York Times and you have many years of archives that maybe don't have links any more then you can build a list of all the pages on your website and save this text file of links in a special format called XML, name this XML text file something like sitemap.xml and then tell each search engine where to find this file. You can also leave a note for the crawlers in a special file on your website called robots.txt like this: Sitemap: http://www.yourwebsite.com/sitemap.xml and the next crawler that comes along will self-discover that you have an XML sitemap, read it and crawl you site. It may choose to use the sitemap for its crawling and it may just pass that along to another crawler -- there is a limit to my knowledge and this is crawler specific thing. All that a sitemap.xml file contains is a list of all of the web pages you want to put in the file, a link to each page and a few other notes about the link to help the crawler understand each page's importance, the last time it was changed and importance (to you) of this page on your site. The main way that an XML sitemap will really benefit you is if the pages you include in it are not currently being found by the crawlers and thus they can now discover your undiscovered pages.

So, if you are an e-commerce site you should have a list of all of your products, product groups and any special information listed in your XML sitemap file. You should also ensure that you have an HTML sitemap somewhere on your site that real human beings can use to find products on your website, because as you have learned here, your pages will not automatically rise to the top as soon as you deploy an XML sitemap but your customers will reward you with more sales if they can find the products they want on your website once they get there. Also, as a side benefit, the more human friendly your website is the more the search engines will reward you.

One last thing, creating an XML sitemap file is easy and your e-commerce system should do it automatically which means you probably don't need to hire someone to create one for you. If your software doesn't create one automatically then you should either buy my e-commerce software which does or use one of the many free tools available online to do it or if you are really bored create your own file and put it on your website and link to it from your robots.txt file.

Have a question about this article, e-commerce or the web? Send me an email and I will try and work that into a future post.

Online shoppers are lazy, stupid and mean...

2009-02-11T17:06:00.000-08:00

There, I said it, now lets figure out what you can do about it and how this impacts your ability to sell on the Internet.

Jim Pryor, the brother of Michael Pryor, the co-founder of Fog Creek Software, is a New York University philosophy professor who according to Joel Spolsky, the other co-founder of Fog Creek Software, shows The Matrix in class to illustrate points about Epistemology. Jim's article entitled Guidelines on Writing a Philosophy Paper is one of the best tutorials on any kind of writing and so although Joel and others have used it to illustrate how to write blog posts and better software I'm going to use it to assert that this same principle applies to online shoppers. From Jim's paper:

Pretend that your reader is lazy, stupid, and mean. He's lazy in that he doesn't want to figure out what your convoluted sentences are supposed to mean, and he doesn't want to figure out what your argument is, if it's not already obvious. He's stupid, so you have to explain everything you say to him in simple, bite-sized pieces. And he's mean, so he's not going to read your paper charitably. (For example, if something you say admits of more than one interpretation, he's going to assume you meant the less plausible thing.)

So, you may be asking yourself, yes, but Chris, what about your signature, "How many more orders will this get for me?" question? Well, lets analyze the three pieces as they apply to online shoppers:

Lazy: If someone lands on your website and can't immediately figure out how to do what they set out to do then they will leave because it's too hard and there are too many other choices out there.

Stupid: most visitors to your website know less about your products and services than you do (unless they are your competitors) and also don't know how your products are categorized so unless you make it simple and straight forward for them to find what they want in the language that they understand then they will be confused and leave.

Mean: Once a visitors has a poor experience on your website they will always think that your website is awful and they will tell their friends directly and some will even tell the whole Internet (via online reviews) and this will hurt your business.

Okay, so that's the bad, so what can we do to prevent or at least minimize this situation (and get more orders too)?

Go and read, Don't Make Me Think: This is a great book on how to design your website so that people can find what they want quickly and efficiently and thus satisfy their lazy tendencies. Keep it simple and make sure that your site is quick to load so they don't get frustrated waiting and leave before they even get a chance to like your site.

Ask someone else to review and test your site: Don't only ask your staff if they think your site is great, because they will lie or just miss things like you would, but instead ask people who are potential customers if they understand what you are saying on your site and how your site is layed out. For example, if you display technical information about your product then offer them a glossary or help information on what these terms mean or even better, blog about these terms so you get a better page rank and some free search engine love. Also, are you organizing your products on your site the way your customers shop or the way you order them from the manufacturer? Think about how people would buy from a physical store and walk through the process out loud to develop a use case scenario that you can then play back against your online store to see if it makes sense -- I guarantee you that this will not only make you laugh, and then cry, but it will also highlight absurdities that you never realized before.

Measure and correct, quickly: If you don't measure what visitors are doing on your site and you expect them to tell you then you are setting yourself up for failure. Customers won't tell you when they are displeased so you need to observe their patterns and see if they are exiting too quickly or bouncing off a page or failing to make it through checkout. You should also monitor for negative reviews and set up a Google Alert to watch for them automatically. Then when they do happen you ideally want to respond both directly to the complainer and also to the issue itself so you don't have this issue again.

I hope this helped you find at least one thing wrong with your current site or process and if so I would love to hear about it, so email me (chris@xmodus.com) or comment below (you can call me too I'm extension 101).

2008 XModus Summer Students

2009-02-11T03:02:00.001-08:00

This is a post that I wrote in September 2008 but did not post until today -- no reason, I just forgot to post it.

It is fall and that means our summer students have returned to school, suddenly we've lost testers, developers, designers and marketers, all at once. Our students are considered full staff during their time with us, okay so they don't get paid parking (we only have a few spots!) and they don't qualify for our health benefits, but otherwise they get all the rights and responsibilities of regular staff.

Like any parent sending your kids back to school in the fall there are mixed emotions, they are growing up and away and at the same time they are freeing up space in your life's schedule.

The Good

lots of work done
great discussions on software, business, politics, religion and life
we all learned new things, they learned about coding, software and business and we learned about sky diving, cars, music and life
we all had a lot of fun with minimal drama

The Bad

lots of extra work to manage the students, their work and their needs (like computers and chairs for example)
some of our students got their first lessons in responsibility and growing up – welcome to adulthood
the extra staff really taxed our management abilities and caused a lot of extra HR-related work and paperwork
we ran out of chairs, desks, computers and even neared the limit of our office space – oh and washroom space too

What we did this summer

wrote lots of code
tested lots of code
website redesign vote
daily scrums
birthday cake
leaky roof
lack of parking
extra water (and toilet paper)
used up our network ports and virtual machines

Some New Stuff

Indian food (vegetarian)
lots of sushi
real software projects with real responsibility

Our goal:

come back for a second co-op, or
come back full time after they graduate
tell their smart friends about how great it is to work at XModus
do better in their school work because of the experience and training they received during the summer
tell stories of their experience that inspires others to do better – the pass it along mentality
provide a job reference for all of them
do real work on real projects for real customers and see the results of their efforts

Their goals:

learn new stuff
show everyone how smart they are
meet new people
maybe get a job in the future

Apple Hour
We developed an "Apple Hour" @ 3PM everyday where anyone who brought an apple would get together at 3PM each day and eat their apples and talk about stuff... think of a smoke break with apples -- maybe you had to be there.

School Breakdown

43% – BCIT
29% – UBC
29% – SFU

Studies

29% – Business
29% – Engineering
29% – Software Development
14% – Database Development

Debates:

politics
religion
vegetarianism
what actor they would like to have play them in the made-for-TV movie about their summer experience at XModus

Sad good-bye, we'll miss all of them and we wish them good luck and hope they come back some day.

My Welcome Speech:
Welcome to XModus, we have selected you to join our team this summer and we are excited that you chose to join us as well. Right up front, this isn't a co-op term this is actually a job interview, because if we like you we will ask you to join us, if you want to come back, or we will provide you with a job reference for future employment. But if you don't give it your 100% or worse you goof off we'll fire you (even co-ops) or we will choose not to offer you a job reference for your future jobs. We're picky and we only pick smart motivated folks that we know can succeed in our environment so we're here to help you realize that potential and when it all goes well it's good for everyone.

Looking for a summer job with XModus?
Think this sounds like the place you want to work this summer then send me an email (chris@xmodus.com)... and tell me you saw this on my blog... it won't sway me to hire you... or will it...

Code Blue - Technical Support for Website Heart Attacks

2009-02-11T02:42:00.000-08:00

This past summer we were courting a prospective client and her site went down unexpectedly -- while she was still with her old provider. I had sent her an email before this had happened and when she tried to respond back she could not, so she called to talk to me but I was not in so she chose another extension on our phone system and ended up accidentally talking to one of our summer students, Aidin, instead. He listened to her problem and then did something interesting... he created a technical support ticket for her and set about finding out what was wrong with her site. Let me give you some context. She had expressed an interest in our Rebus software and we had scheduled a demo of it and that was it. So, why did he create a tech support ticket for her? Well, two reasons, the first is that her current provider was not responding to her phone calls and so she was quite desperate and the second reason was that he, the student, realized that with her site offline so too was her business and thus her company was in effect having a heart attack of sorts and needed immediate intervention, which is typically called a Code Blue, at least in all the cool hospital-related TV shows and movies.

So, the obvious question is, "What the heck does this have to do with me and why are you blogging about this other than to brag about your awesome summer students?". Well, stick with me for a moment as I explain how we as a company have come to adopt this notion of a Code Blue event when our customers experience a disruption of any kind and the process we went through to train our staff to understand the seriousness of this to our customers.

We are cross-training some of our staff for a new role we call QFE, quick-fix engineer. Their job as QFE is to evaluate the really technical technical-support requests and during our discussions we realized that we needed a way to express the level of urgency for our highest priority events and came to realize that they are 'like a heart attack' which lead us to this page Heart attack symptoms: Know what signals a medical emergency. You see, as a business owner myself my heart gets going as soon as any one of our customers is having a problem, the more severe the problem the faster my heart races, but we discovered that I suffer from a condition called the "curse of knowledge", see Chip & Dan Heath's book Made to Stick, in that as an owner myself I intuitively understand the level of threat that another owner feels when there is a support issue that is impacting their business in any way. This is something that I understand but not necessarily something that a support person reading a support request would understand and act accordingly.

When I tried to describe the threat level, as I call it, to the rest of my support folks I started by telling them that when our e-commerce system fails to perform as the customer expects then they take it as a personal attack on themselves, which then means that by proxy you are also attacking their kids, their family and all of the kids and families of their employees, their suppliers and all of their friends... some folks got this and others just thought that I was being overly dramatic. This lead us to change the metaphor to a heart attack as it is consistent with our software being "the heart of our customer's companies". This new metaphor allowed us to convery the seriousness of the situation by creating an imaginary scenario that they had to take a sick family member to the hospital because they believed that they were having a heart attack and that the 'support' folks there ignored them -- okay, this is typically what appears to happen, but really these folks are triaging you and unless you truly are having a heart attack (or other equally serious problem) then you get placed into the queue for the next available person and wait your turn.

Yes, but what does this mean?
If you call XModus technical support and you have an emergency, a real heart attack type emergency, something that involves your order processing at some level and is thus blocking orders from coming in or going out, we have a very clear procedure that we follow to ensure that you are kept in the loop throughout the triage process and through to the resolution. We visit an internal Code Blue procedure page that spells out our procedures, we then immediately contacts our internal folks, create a tracking ticket, and then we contact you at regular intervals (depending on the actual severity) to keep you up to date so you can keep your staff and shareholders up to date and so you aren't more stressed than you have to be during this event.

"How many more orders will this get for me?"
So, you may be asking yourself, yes, but Chris, what about your signature, "How many more orders will this get for me?" question, to which I suggest for this situation we look at the question in a different way -- how many orders will this save for me and how will this ensure that during a crisis that you, our hard fought for customer, remains confident in our ability to resolve your critical issue as quickly as possible, that we document the cause so you can evaluate your options to avoid this in the future, and we provide you with steps to either prevent this from happening again or a resolution that this won't happen again. I would also suggest that if you don't know the emergency procedures for any of your critical suppliers right now, typically called your service level agreement (or SLA), that you find out during a non-crisis time so that when the time does come that you are confident that your emergency will be dealt with as quickly as you need and not when someone gets around to it.

We could have let the prospective customer at the start of this story sort out her own problem as it wasn't really our problem but that would be like walking past someone on the street when you know you could probably help them. This brings me full circle back to Aidin to wrap this story up. So after creating the ticket he grabbed a network engineer and they jumped on the problem and discovered that the domain registration had been allowed to lapse and that had caused the domain to be suspended with her registrar and a quick visit to the registrar's web site and a renewal of the domain and presto the site was back online. As you can image she was very impressed with our service and kudos to Aidin for recognizing the urgency of the situation and acting accordingly to resolve a critical situation.

She is now a client of ours and our support folks better understand the urgency of these critical customer support calls and act on them accordingly. The only thing that we're missing now is a cool blue light that turns on when we have a Code Blue situation to really add to the drama... maybe one of our students this summer can fix that.

Link Exchange Sites - Search Engine (Scam) Nirvana!

2009-02-04T00:21:00.000-08:00

So from my previous post about how Google's core algorithm is all about linking to and from websites then you are probably thinking, Chris, how can I get a bunch of free links to my website that will boost my Google position? Well funny you should ask and re-read my previous post for two free ways to do that, but today I want to talk about one seemingly perfect way to do this, link exchanges. In fact I want to share with you an email I received from a friend on this very topic and the research and the research that led me to discover that link exchange websites can actually be scams designed to boost the Google score of someone else and rob you of your Google score and even possibly land you in Google jail for trying to manipulate your Google score.

Here is how it all started:

Chris,
We get inquiries to link to sites on a daily basis, the latest one is below. What are your thoughs about two-way linking with companies who are interested in exchanging links? I think we may be missing a golden opportunity here!

And here is the original email that she references above, note that I have left the poor grammar, typos and weird spacing as is and have only removed the names to protect the innocent and block the scammer. I have changed the product to 'flashlights' in this example.

*From:* Links Specialist
*To:* webmaster@website.com
*Subject:* Link Exchange Request

Dear Webmaster,

I am Promoting a websites related to Flashlights. As you know 3 Way linking increases the Ranks and Traffic to the site bringing clients i will look forward for this beneficial link exchange . If interested please add my below given details on your site and i will add your link ASAP. If you are interested in link-exchange then I can add your link here:

http://www.website1.com/shopping.html (You can choose any category)
http://www.website2.com/shopping/shopping.html

*The details of my site are as follows:*
Title - Cheap Flashlights & Bulbs
URL - http://www.cheapflashedlights.com/
Description - We offer a huge selection of Flashlights for all the top brand names. (This is where they included information from my friend's website too)

As you know backlinks helps in generating more traffic to our sites as well as achieving higher search engine rankings.

If you own any other sites for which you are willing to trade links, please let me know. I'll be happy to add your listings to my site immediately. I'll be glad to hear anything you have to offer.

Best Regards,
Links Specialist
webreseller192@gmail.com

So, when I first read the request from my friend I thought she was referring to sharing with someone else that she knew but when I read the forwarded email I was suspicious and here is why:

1. The "From: Links Specialist" was too generic and it was from a free email provider, in this case gmail.

2. The "Dear Webmaster" immediately jumped out as SPAM.

3. The funny spacing and the way things about my friends website were sprinkled throughout made me think that this was an automated email that was merged with a list of websites from somewhere.

4. Finally, the person who sent the email signed it not with a name but with that silly "Links Specialist" tag again. If they had said, "Joe Smith, Links Specialist" then it could have passed as a legitimate title but not as their name.

So I did what I always do for my friends, I investigated and here is what I found:

Hi (friend), this particular email is a scam. I followed the links in the email and once I went below the first page it was all bogus and even potentially damaging. If you are getting legitimate link requests from partners or sites you can actually verify are legit then you can add a content page to your website, add a link from the footer and there is your links page -- but don't expect that it will zoom you to the top of Google unless it is from an authority site like one of your manufacturers.

Here is some info on link exchange scams: (I did a Google search: "link exchange scam")
http://www.geeksaloud.com/8/the-reciprocal-link-exchange-scam

I even found a page on how to write and send the very email that you received called the "Guidelines to a perfect link exchange scam"

- Chris

It was particularly 'funny' finding a web page on how to create and run this
particular type of scam. Needless to say my friend was grateful and she did not pursue it any further. One final thing, please read my previous post on how Google's main algorithm works and if you have any questions about link exchanging please send me an email chris@xmodus.com or post a comment below.

How Google Ranks Your Web Pages

2009-02-04T00:01:00.000-08:00

Have you ever heard the expression, "everything I need to know about life I learned in kindergarten."? For this post my version is that everything you need to know about how Google ranks web pages I learned during my Wednesday night volleyball games. You see, I play recreational volleyball on Wednesday nights with a group of Ph.D. students... I'm not a Ph.D., I don't play one on TV and we don't typically talk about their studies but I live near a university and my team just happens to have a couple of Ph.D. students on the team. So, during a recent conversation they were telling me about how they need to publish articles on their area of study on a regular basis and one of the critical pieces of any scientific research paper is a list of the references at the end of their papers. It turns out that the folks that wrote the Google search engine took this principle of referencing and used it as the basis of the algorithm that ranks pages on Google. Okay, I'm sure they have more than one ranking algorithm now but for this post here is how it works in academia and also how it is spelled out in the original Google Page Rank Algorithm thesis paper that the two founders of Google wrote.

Here is how it works:
1. Suppose you write the first and only article (web page) on a specific topic and Google finds your page and sees that you are the first and gives you a score of 1 (out of 1). So anyone that searches Google will only find you and you will be pleased.

2. Someone else writes an article on the topic and doesn't reference you and Google finds their paper and sees that you two share the same topic but do not reference each other so they give you both a 1, but now it is out of 2.

3. A third person writes an article on the same topic but this time they reference your article and Google finds their article and they assign a score of 1 to this third person but then takes a fraction of their 1, lets say for the sake of argument they take .25 of this one, and they assign it to you as the reference. So now the scores are 1.25, 1.00, and 0.75 (all out of 3), so when someone searches for this topic you are still the first entry and you are pleased.

So the page rank essentially works that the more people that reference your site, making you an authority on the topic, the more favorable your Google score on this topic and thus the higher you place for searches on this topic. Now, again, I'm simplifying this to highlight the core principle behind how Google works so you can see what Google really values and hint it's not META tags.

So this was all good until someone figured out that since Google was counting references (links) that they could build a series of bogus web pages that simply referenced their page and that this would boost their search engine placement. This then lead to people creating entire websites devoted to the selling of links for this very purpose. This then lead to consultants claiming that they could get your website to the top of Google search results simply by having them create links on all of the many websites they owned or controlled that sold these links and just simply placing you on their website, typically for a fee. Sounds like a good idea and typically it was cheap... that is until Google discovered that people were gaming the system and they slammed the door on these folks and all the sites they referenced and one day your Google traffic was great (and free) and the next day you were dropped from the Google index (true story). Google is ruthless towards people who they feel are manipulating search results, just ask the German car company BMW.

I will have another post on link sharing shortly, but back to the post.

So, if Google wants people to link to me and the more people that link to me the higher I rank on Google then I simply need to find people to link to me and I'll be at the top -- so Chris, where are these people? There are several sources for these and it's important to choose the best people, the ones that Google trusts and it's important that they link to your website and pass you some of their 'Google Juice' when they do it. There are several sources for links but here are two easy and free sources:

1. Get listed as a retailer on the website of the manufacturer of the products you sell. This has two benefits, one is that people who are looking for these products will visit the manufacturer's website and implicitly trust you because you are recommended by the manufacturer and second the manufacturer will typically be seen by Google to be the authority on this product and thus their Google score will be higher than your Google score and so you will gain from their reference to you.

2. Blogs. Google likes blogs because blogs help people and so Google typically gives blogs higher rankings (there are other reasons but lets keep this simple). Now before you tear off and start your own blog and link to all of the products remember that Google wants to see real links, so for example one of my friends discovered that someone on a blog for a TV show discovered that he sold the product that was used in the TV show and linked to my friend's website -- at no cost to my friend and when we searched Google we discovered this blog post and the link to his site. Ideally you want people to start talking about your website and the products on it in a natural and human friendly way and then once others find this useful they too will blog about it and as the saying goes and they will tell two friends and so on and so one and soon, if you are incredibly lucky, this viral marketing will boost your Google score and will also expose you to a whole bunch of people for free -- and the implicit 'trust' given to you by the bloggers will even help with selling your website to these visitors.

So, I hope this helps clarify that most of the weight that Google gives to your website and its pages has less to do with the font size, the page color, the META tags, the TITLE tags, and most other SEO voodoo suggestions and more to do with the folks that link to your site and pass their Google score to you.

Have a question or comment, email me, chris@xmodus.com.

SEO Compensation Conflict of Interest

2009-02-03T23:54:00.000-08:00

I recently received a phone call from a friend asking my advice on engaging an SEO consultant to help them with their marketing activities. In this particular case the consultant had told them that they would increase their sales on Google and that they would clean up my friend's Google campaigns. My friend was skeptical but willing to entertain a quote from them. The consultants returned with a simple proposition, pay us 12% of the money you spend on Google each month (my friend had already told them how much he spent so it was an easy calculation) and we will clean everything up and increase your traffic. So, if you've been keeping up with my posts of late, especially this one you will know that I always ask, "How many more sales will I make if I make this investment?" but in this case there was something else wrong with this picture and I've seen it before, let me explain.

A few years ago a friend of mine was engaged in a similar arrangement with a similar company and they even offered to do the work for a percentage of the marketing spend. At first this seems like a good deal because you think you control the spend today by simply putting in a budget and in this particular case Google never actually used all of the budget each day so the total spent on Google at the end of the month was much lower than the budget. This is because my friend was not actively looking to spend that much money each month but instead was willing to spend at most that amount with the implicit understanding that really they only wanted to spend the maximum when the sales were adequate to support that level of spend. Now on the other side of the fence the consultant was looking at the maximum budget on Google and doing their calculation on that number, so not the current actual spend but the potential marketing spend... guess which one is usually a lot bigger. Now to make this matter worse for my friend, the one with the existing engagement, he wasn't tracking this consultant very well and so when the consultant told him that he was generating more sales and my friend saw that his sales had gone up a bit he trusted that it was because of the consultant. But here comes the rub, after the first few weeks the consultant came back to my friend and told him, truthfully, that my friend needed to spend more to make more and that there were missed opportunities to grow his business and thus my friend needed to
increase his marketing spend to realize these opportunities and so he did. What my friend didn't immediately realize is that he had also just increased the amount of money that he paid this consultant as well. Now, this would have been a great deal if in fact the consultant was bringing in actual sales and the profit for these sales was positive, but in my friend's case they were not.

So back to the original reason for this post, my view is that this relationship is a conflict of interest because the person who controls the decision to increase the marketing spend is getting compensated based on the amount of the spend so their natural tendency is to increase the marketing spend and to find ways to do this. This however is possibly at odds with your reason for hiring them which is not to increase your spend but to increase your profits and the two are not the same. You want to find the best set of marketing activities that costs the least and generates the most profit and this does not mean that you do this by spending more on marketing -- though that might be required.

Have a question or comment, email me, chris@xmodus.com.

Traffic Cowboys - Driving Visitors to Your Website

2009-02-03T23:43:00.000-08:00

Several years ago one of our customers was investigating ways to increase the traffic to their new website and so they did a web search and came across a company that guaranteed to increase the amount of traffic to their website and they were so confident that they could increase the traffic to any website that they offered a guarantee -- if we don't increase traffic then you don't pay us. Sounds great doesn't it? An increase of traffic to my website guaranteed... there has to be a catch. No, there was no catch, they did deliver on the amount of traffic and I'll talk about how they did it in a minute, but first my annoying question, "How many more sales will I make if I make this investment?" No one asked this question and that's where the traffic cowboys failed to deliver as expected, though they did deliver as advertised.

Here is how I explained this situation to a friend recently who came to me with a similar proposition from a consulting organization. You have a retail store, for this example assume it's a physical store, and you are situated right near a busy street with lots of foot traffic so you assume that your store will get lots of people dropping in, so many in fact that you won't have to advertise. But after the first couple of days you realize that you need to advertise and no sooner do you think this then two cowboys arrive with the promise that they will get you more traffic, guaranteed, all you have to do is pay them for each visitor they bring in -- no visitors, no pay. Seems like a great deal and being the crafty business person that you are you decide to wisely try before you buy. So on the first day of their services people just keep coming in the door and you are pleased that you decided to use their services and you watch as people come in and out of your store but none of them appear to be buying anything. So you go outside and you discover that the two cowboys are standing outside of your store with a detour sign sending customers into your store as they walk down the sidewalk. You are understandably not happy so you tell the cowboys to stop and they do and they come in to your store and are curious why you want them to stop driving traffic to your store and to which you reply that you want shoppers not traffic. Oh, they exclaim, you wanted shoppers not just traffic. Well they guaranteed traffic not shoppers, that's your job to sell them once they bring traffic to your store. They thank you for the money you owe them for driving traffic and move on to the next store.

Now, one more thing that came up during a conversation that I think is very salient to this example and something I hear all the time, but Chris I would notice right away if there was a problem and they have references and they have other customers so these other folks must be getting value from their services and also they wouldn't be in business for very long if their services didn't actually work. So let me dissect this argument because I hear it all too often and it drives the logical side of me insane.

I would notice if there was an issue...
In my physical store analogy you, the crafy business person, were able to step outside of your store and see the problem of the detour sign and you were able to fix it immediately, but what would have happened if you were not at the store and instead your minimum wage store helper was tending the store and instead you phoned in at the end of the day and asked how it went and your cheerful helper said that it was a great day and there were so many people and it was crazy. You might ask them how sales were and you would probably attribute any increase in sales to this campaign -- and if there was no increase you would probably discount it and rationalize that it might take some time, in fact they may even tell you that. If you weren't actually measuring the results of these transactions and weren't looking at them immediately you wouldn't see that there was a problem until much later in the engagement. In fact your sales might be going up because of the cowboys and so you would be happy, unless you started to actually measure the number of sales and the contribution to your margin that they made after subtracting the cost of the two cowboys, remember it's all about adding more to your financial bottom line.

Always verify customer references...
Clever sales people always ask for references as soon as they can and preferably in writing as they know that future prospects will trust other people more than they will trust the sales person and so they get references -- it's Sales 101 (I know this because I just took a Sales 101 course!). Most people trust what they are given and don't check references ... same for job references but that's another post. It might be that they are paying those references and it might be that they are providing a great service and nothing is wrong here. I'm not saying these folks are nefarious, crafty and untrustworthy but I am saying that you always need to check references and verify that the reference is relevant and credible (and not the guys brother -- it's happened to me before).

Most people don't measure...
Guess what, most people don't measure their marketing so if they see an increase in sales after employing the cowboys then they will naturally assume that it's because of the cowboys -- but they don't verify and they don't measure. So, they will have lazy customers who don't measure and they will make money for them and the trick here is that you can't be lazy in a situation like this, you have to measure because when we measured for one of our customers who went down this path they immediately discovered that indeed they were getting more traffic but not more shoppers and so they forced their cowboys to bring them more shoppers and paid based on shoppers until the cowboys dumped them. So this particular argument covers two things, their references may not measure and thus are blind to the actual performance, at least for now, and two, they will continue to make money from all of the folks that are not measuring until they measure and so they will get as many as they can as quickly as they can and either figure out more services to keep these customers or just keep adding and dropping customers until they find something else to do.

So, did you figure out what happened to the customer I referenced at the start of this post? We put in place tracking mechanisms to actually track and measure the performance of their cowboy campaign, actually we did this several months after they had engaged their cowboys, and the results were abysmal. They took our data back to the cowboys and confronted them about it and they, the cowboys, swore that they would improve the quality of the traffic but instead came back to our customer and asked for a reference first and in return they would give them a discount on their upcoming improved visitor generator that was guaranteed to drive not just visitors but shoppers! Our customer declined and instead told them to shape up or ship out and so off into the sunset the cowboys rode.

Oh, in case you are curious, what is the equivalent of a detour sign on the Internet? Well one of the groups that we encountered would buy domain names as soon as they expired and immediately put a redirector page on the domain that pointed to our client's website. Here is the problem though, in one case the domain they bought was for a defunct historical society and so the people that were visiting the historical society were shocked to discover that their final destination wasn't the society but instead an e-commerce retailer selling something completely unrelated to their destination and so the bounce rate was nearly 100%... but not 100% so every once in a while someone would buy something, but since they were paying per visitor the conversion ratio was horrible and so the actual cost at first seemed very low but in fact was actually higher than other methods. So imagine if you went to a store and when you opened the door it was another store and a completely unrelated store at that, would you stay, probably not, unless coincidentally they sold something you needed.

Have a question or comment, email me, chris@xmodus.com.

SEO Voodoo - Meta Tags

2009-02-03T23:34:00.000-08:00

If your SEO consultants have told you that you need to improve or change your shopping sites META tags then ask them this question, "How many more sales will I make if I make this investment?" -- and when they tell you that it is an SEO "best practice" and that they cannot give you an exact number it's because META tags don't directly translate into sales and so you should be asking yourself, "Then why should I care about META tags?"

Maybe your first question to ask is, "What is a META tag?" Good question. Here is the Wikipedia entry and here is my simplified version:

META tags are hidden HTML tags that historically were used to describe the individual web pages on web sites, circa 1997 - 2000 and they look like this:

<meta description="The Big Red Page with lots of information">
The description attribute is meant to give each web page a description that can be displayed for your page. However, this is not the same as the title tag which is what is used to display the title you see at the top of your browser.

<meta keywords="big,red,page,information">
The keywords attribute is meant to provide machine readable keywords used to describe each page on your website. However, modern search engines, post-1999 all use the actual content on your website over these keywords.

Historically, again circa 1997, the search engines encountered a problem, namely that people were stuffing their META 'keywords' attribute with every single word they could think of and in some cases multiple copies of the same word in an effort to increase their search engine positioning, this technique is known as keyword stuffing. This is why even 12 years later people still recommend that this tag be filled with the juiciest words for your website -- and I'm not going to argue that you shouldn't fill in this tag but be aware that the search engines are wise to this practice and don't place much (if any) weight on them anymore.

But let us get back to my original question, "How many more sales will this generate?!" Let me give you the technical answer: probably none. Why? Well, here is the problem, search engines don't use these tags for ranking your site, so putting in the optimal values into your META keywords won't impact your ranking which therefore will not impact your sales, hence you'll be wasting your time (and money). Here is another problem with my question about the sales performance, there is no way you can track this unless you spend even more money and time to do some form of A/B testing where you display (and track) your website with META tags and without META tags and then see which version performs better on organic search. You would be wasting your time.

Now, not all SEO consultants are META tag focused and I invite any of them to send me an email or post a comment if they disagree. What we find specifically with META tags is that retailers are spending too much time on things like META tags that don't impact their sales instead of working to improve their site to maximize their sales and actual generate more profit for their site. The problem is that most general-purpose SEO consultants simply don't have enough information about your business and don't know enough about your industry and products and so they must focus on the general stuff -- like META tags.

So the take away from my rant is that you should know that META tags are not critical to the success of your retail site and that you should be spending your time and money on improving your website for humans first and foremost (more in a later post) and at the very most you should make sure that your META description tag is correct and that your META keyword tags are at least accurate but don't invest a lot of time and money on them.

Have a question or comment, email me, chris@xmodus.com.

SEO Voodoo - SEO survival information

2009-02-03T23:25:00.000-08:00

We've coined a term here at XModus, "SEO voodoo" (SEO = Search Engine Optimization). This is the 'black magic' and techno-babble that we hear echoed from some of our customers when they have engaged external SEO consultants. We feel that SEO optimization is critical to the success of all online businesses but there is so much misinformation and so many unmeasurable "best practices" that we thought we would talk about them here on our blog and give you, our dear readers, the information your need to avoid or at least understanding the truth behind their (billable) recommendations.

One central question that you should always ask any consultant, SEO or otherwise, is how many more sales will this generate or what is the measurable return on investment from this activity. We will try and help you identify measurable activities and also unmeasurable activities that we find time and again are touted as cure-alls for your website.

One more thing -- everyone's goal is to be the first organic search result to appear in any search engine and so this makes all online retailers susceptible to folks who claim they can boost your position so beware. That said, search engines value human readability over tricks and schemes so the focus of your efforts should always be to make your site easier for people to read (and use) which will have the double effect of helping your customers and also helping your search engine positioning. We will cover real techniques and mechanisms for helping you improve your search engine score in future blog posts and feel free to email me, chris@xmodus.com, if you have specific questions you want me to answer or talk about here -- SEO or otherwise.

Using Google Alerts to keep tabs on your websites and products -- for free

2009-02-03T23:23:00.000-08:00

Here is a simple (and free) tool from Google called Google Alerts that you can use to keep tabs on:

1. the use of your website name
2. the use of the name of your top products
3. the use of your competitor's names
4. negative reviews of your site, especially in blogs

The best part is that it's easy, free and you can turn them off whenever you want.

Have a question or comment, email me, chris@xmodus.com.

How to Kill a Sales Relationship

2009-01-21T23:47:00.000-08:00

You know the old adage, "it takes a lifetime to build a relationship and only a single moment to destroy it"? Well I received an email recently from a company that we have been doing business with for a number of years (lifetime) asking me if it was okay if they stopped paying us our annual affiliate payment (single moment). Here is an edited version of the email they sent to me, the names have been changed (and a couple of grammar fixes too):

Subject: Hi from (Company)

Hi Chris,
I hope you had a great holiday and new year. I am starting to get a bit of heat from my management regarding the annual payments that we make to you guys for the (company name) account you signed up with us a few years ago.

Can you provide me with any insight as to what your company does to ensure that this client (company name) remains a client and let us know whether it is okay with us just cancelling our contractually agreed upon annual payment to you?

Thanks,
Skippy
Manager - Business Development

First of all, I added the "contractually agreed upon" part and second, thank-you Skippy for starting this email with "I hope you had a good holiday" as I found that soothing while I gritted my teeth and started reviewing our contract. I also appreciate continuing with "because I'm having a problem and I think the solution is that you (Chris) agree not to get paid by us anymore" even though you (Chris) brought us a great customer that has brought (and continues to bring) our company a large sum of money annually. So I suspect that in fact Skippy is getting heat from his managers, something along the lines of, "Hey, these XModus folks are getting paid this amount annually and what have they done for us lately?", and also possibly, "Where can we cut our expenses this quarter to make things look good, at least in the short term?" (okay, I added the 'in the short term' to emphasize my point about destroying the relationship).

So, first I think that email was a poor communication option for this type of 'Dear John' letter and second, who in their right mind would agree to stop getting paid for work that they did years ago that they were told, contractually, that they would be rewarded for... that and it's 2009 and we're in the middle of a recession so I'm not all that keen on giving up any cash owed to me.

Here is how I would have done it differently:

I would have sent an email asking for a time to talk about new opportunities and also to review our current relationship, maybe I'm not happy with them right now and he doesn't know that. Then I would have called and talked to him on the phone (or in person if possible) and inquired about the relationship that I was having with our mutual customer. I would have used that information to determine if there was a next step as drastic as ending the current relationship or if no action was required. If the relationship needed to be changed I would have sought out alternatives and then if there was no alternative then I would have played the ultimate card of ending the relationship. I would not have brought up the termination provisions in our current contract (something he did in a subsequent email).

Here is what I did or plan to do:

So my response back to Skippy was to ask to speak to his manager directly, to which I was rebuffed, for if in fact they feel that they should stop paying our annual fee then perhaps my customer would entertain switching to a competing service that Skippy's company provides -- and I suspect that said new company would probably be willing to pay my company a similar annual payment that Skippy is suddenly getting heat over. I suspect that the heat that Skippy is feeling today is going to get a lot hotter once I get his folks on the phone and point them to this blog post and discuss how we can help them stop paying us by simply removing this troublesome account from their burdened hands and moving it over to a competing service that does not feel as encumbered by this situation. I have, for the first time in a number of years, reached out to their competitors and started a conversation with them and this could ultimately mean that Skippy's single action could cost the company he currently works for a lot more than the small annual fee that we receive from them each year.

- Chris

When Software Updates Can Hurt Your (User's) Feelings

2009-01-20T21:47:00.000-08:00

One of the reasons I use the Mozilla Firefox browser is because it supports these great tools called Add-ons (http://addons.mozilla.org). Since I run a software company that builds web software we use a great number of Firefox Add-ons to make our jobs easier and so on January 6th, 2009 when I started up my browser and it told me that one of my Add-ons had an update and prompted me to update it I didn't give it a second thought – why not, all of my previous updates had brought bug-fixes or new functionality to these great Add-ons – but not this time. This time the Add-on did something both unexpected and unpleasant – it changed my browser experience without my permission and suddenly I felt stupid (that's bad), exposed (more bad) and finally angry because I had let someone onto my computer in such a laissez-faire and careless way. In essence I felt that I had been embarrassingly sloppy by letting this supposedly innocuous software Add-on onto my computer.

So what did this patched Add-on do to make me so mad you ask? Well the Add-on in question is called "Fast Dial" (https://addons.mozilla.org/en-US/firefox/addon/5721) and it provides a grid view of popular pages that I see when I start a new tab in Firefox, something I do all the time. Normally when I start my browser I click a new tab and then choose from the quick select options provided by Fast Dial, but not this time. This time when my browser started a new page appeared, with a search box and several tabs that were not mine and then I noticed that my search bar had changed to a different site. My first thought was to shut down my computer immediately and quarantine it because I had been hit by a virus somehow... but then I stopped for a second and looked at the Fast Dial page and realized that the pieces of the page all seemed to be related and that the new search option was related as well. So I quickly jumped onto the Firefox site and over to the Add-ons section and to the Fast Dial page and there in front of me was a litany of complaints and angry comments from folks just like me who had applied the update only to discover that their world had been changed – without their permission and with no advanced notice.

So I read through the comments and soon realized that I was not alone in both my shock and displeasure and that what was once a very highly voted Add-on was suddenly seeing its rating plummet from 4+ stars down to 2+ stars and the comments were almost vitriol. I also noticed that over the next three days the number of votes with 1-star rose from 77 for the whole of 2008 to more than 500 within the span of three days – so one simple update destroyed an entire years worth of good ratings... one (surprise) update!

We had a similar situation about a year ago with our Rebus product. One of our customers had pointed out that one of our reports could be enhanced by adding additional costing data to an existing report. We thought it was such a great idea that during another update to our software for our other customers we slipped the updated report in, free of charge, and were eager to hear all of the accolades that would surely follow... the roar was deafening. Our users were so upset that we had changed the formatting of their report, the report they read every day, the report they export to Excel and use as part of their monthly report, the report they need to know how their business operates every day. We were shocked; as I'm sure the Fast Dial developer was too, at the outrage and angry comments that followed. Here we had thought that we were helping our users and instead we had surprised them and impacted their day to day operations unintentionally and they lashed out. We rolled back the change immediately and created a new report for the lone customer who had requested the change originally and released the new report to our other customers as a new (and free) report.

So I didn't write this to criticize the developer who created Fast Dial, up until January 6th, 2009 I was pleased with the (free) fruits of his labor, instead I wrote this as a cautionary parable to other developers. We learned from our mistake that even with the best of intentions that users need to be aware of changes before they happen and hopefully also that they want the change so that when the change arrives at least they are prepared for it. Even better, we now release upcoming patches to our staging environment for customers to preview before we patch their systems and they are free to comment on these changes before we roll them out and so far there haven't been any negative surprises since.

Oh, and I did uninstall the Fast Dial Add-on and have no intention of re-installing it. I also uninstalled a bunch of other Add-ons that I no longer use and have reverted back to using my bookmarks for now and reading the update notes on all Add-on updates before I perform the update so there are no more surprises.

Vancouver Power Outage - No Downtime For XModus Hosted Customers

2008-07-17T00:01:00.000-07:00

Hosting companies, specifically co-location facilities, provide three P's -- Power, Pipe and Ping -- and they promise to ensure that they are available 99.999% of the time (the five nines). This week Vancouver, British Columbia, our home town, was the victim of a power outage that disrupted power to the core of the city for nearly three days. One of our data centers is located within a city block of the main outage but we only found out about the outage when one of our employees received a telephone call from his girlfriend that her company had no power and so they had all been sent home. We were shocked! None of our monitoring alarms had sounded -- was something wrong with our monitoring at such a critical time? So we immediately checked all of our network connections and servers and everything was online and operating normally. There were no alarms because nothing was wrong at our data center.

What will our customers think?

Expecting the worst we immediately called all of our hosted customers to let them know about the event and to tell them that we were prepared to move them to another facility if the outage were to impact them. They were surprised and relieved that although there could be a disruption to their business that we were prepared for it and that we had a plan if the worst were to happen. We also called them because we realized that these websites run these companies and we certainly didn't want our customers to find out about the outage by having there websites suddenly off-line and then having them all try to call us at the same time to find out what was wrong. Our Rebus software is the life blood of these online retailers so having no power, pipe or ping on a Monday, typically the busiest day of the week, would mean loss of revenue because they couldn't take orders and loss of productivity because they couldn't pick, pack and ship them either.

It didn't happen, we stayed online.

Our hosting company is, Navigata. They routinely send us notifications about emergency power generator tests (about once a month or so) and we just file them away with the expectation that when a day like Monday, July 14th comes and the power goes out all around them that they will stay online and so will our customers... and they did and we did and our customers did too.

What did we learn?

We learned that our hosting company really does test their emergency power systems and that they work! We learned that our customers appreciate being kept in the loop during a crisis and that having a disaster recovery plan in advance means you execute on the plan without thinking and that means you get to go home at the end of the day and not worry that your customers may be offline at any time.