…and around it goes

This is covered completely on our VPN pages, but I want stress it clearly that PPTP should only be used as a last resort.  Even then, it should be used in conjunction with SSH tunneling as additional protection.

PPTP has major flaws.  It’s encryption uses the password as the key and it’s datastream carries a retrievable password hash.  To make that perfectly clear, someone can take your password out of the datastream  and decrypt your traffic.

Granted, there is more to it than that.  They must somehow intercept your traffic, but while complicated, it is not impossible (we have seen and stopped attempts at arp poisoning attacks).   They also must be able to crack the encrypted password hash.

There are tools to crack this.  However, a very complex password will take eons to brute force.  A common word, even if you replace vowels with numbers (this is too common people), may be cracked in minutes.

OpenVPN is subject to none of these weaknesses.  It uses very strong certificate based encryption (blowfish).  Even if someone does intercept your traffic, they can gain nothing from it.   Use OpenVPN over PPTP.

I’d really like to thank you all for your patience with our issues last week.  Things just came at us one right after the other.  The issues have all been resolved and we’re now back to normal (knock wood).  I’ll now work again towards getting the VPN service released, this obviously delayed things there.

I’d like to thank those who sent the kind e-mails.  I also smiled when a few of you mailed and said “What issues?  I haven’t noticed anything.”  Thanks.  The webmail server has been moved back to it’s original network and ip address, which should have been (and seems to have been) virtually unnoticeable.   All seems to be purring (again, knock wood).

This doesn’t affect any services at all, but I had to mention it.  Now the security cam DVR has a board failure. Fortunately, have a spare, but come on.  Stop the failures!

The attack as increased to such a level we had to abandon the network under attack until it subsides.  The webmail server has been moved.   Everything is currently up and functioning.

The frustration I have over this is immense.  I just don’t get it.  I spread everything out.  Proxies are in a different datacenter than mail, hosting is spread across datacenters (especially dedicated servers and services, you guys haven’t seen a glitch, it’s the public subscription service taking the beating), and mail is set up with four lines, divided so that webmail resides on one net and incoming pop/imap/etc on another so that even if a line goes out, another is there to take up the slack.

This is so that when one service has issues, the others do not.  Except this week, they have all had issues.  One right after another.  I haven’t really slept more than 7 hours in the last four days.  This latest attack is brutal.  I had to abandon the  network involved and move the webmail/sending SMTP to another.  Everything is currently up right now and on a brighter note the proxies/VPN service is clearing up and speeds increasing.

I have received a bit of e-mail over this, a lot of it very irate.  I understand.  I really do, but I will never let this service stay down for long, no matter what the bad luck gods decide to throw our way.   I have also received a number of very supportive e-mails.  Those meant a lot to me.   I thank you for your patience and support and I will return Cotse to stability!

Today a new twist, the webmail and sending SMTP network came under attack.  The result was many lost packets and periods of unavailability/slowness.  I think I got it cleared up.  Still monitoring.  This bad luck streak has to stop sometime.  I keep telling myself that, but every day is something new this last week and a half.

After writing that last entry I went back through this blog and looked at some past entries.  In doing so I noticed a 2008 entry for one of my “bucket list” items.  It was to happen in 2010.  My solo ride across the country on my Valkyrie.  It’s 2010.  That isn’t happening, even though I believe vloging it while basically running the service from my bike would be decent PR, just cannot swing it.

This time it started with human error and a domain expiration.  After that was all cleared up it progressed to a hardware failure.  After that was resolved, a DDoS attack started (we see these a few times a year, usually the proxy network).  After that was cleared up, back to hardware, but this time not ours, a router upstream of our proxy network and out of our control seems to be dropping packets and all we can do is wait for the service responsible to fix it.

I understand this steady stream of issues the last week or so is making us look bad.  It really pains me, this service is far more than just employment to me.  This service is a passion that just happened (well, for values of “just happened” equaling a lot of work) to start paying bills. I originally started it as a free service for one reason only, I believe in privacy.

I did then, and after years of running this and seeing what I have seen, I believe in it even more.  I had to start charging because it began to cost so much to run and the security business I started that was supporting it failed.  In addition, it was taking so much of my time that I had to be dedicated just to it.  And it just grew.  But it is still a passion, not a job.

In case there is any confusion, I’m not one of these high paid “CEO/President” types.  In fact, I don’t even earn anywhere near what I would earn using my skills working as a sysadmin/developer for someone else (and with it demonstrated in what I built with cotse, I’d think finding employment at a CIO level wouldn’t be difficult).  Nearly everything this business earns is turned to growth, support, and new features/enhanced features.

Unfortunately, while we have grown substantially since our beginning, we are not Google nor Yahoo and we certainly don’t have billions to be able to afford so many hotswap devices and hotswap networks that you won’t notice when a machine or router goes bad (and even those big guys still have issues a few times a year).

While they got millions in investment, we boot strapped and did not take a dime of investment (although if the right fit appeared now, I’d consider it to help break out of the niche and make it more mainstream to be private).  All we can do is work under the constraints we have and fight the fires as they ignite and extinguish them as rapidly as possible.  As we grow, you can be assured we’re investing in the business and redundancy, nobody here is currently getting rich from this, we do it because we strongly believe in privacy.

This string of failures hit the full spectrum, human error, hardware failure, attack, and back to hardware failure, one right after the other.  I’m feeling burnt out right now.  It’s a never ending battle to keep everything functioning smoothly, and quite frankly, we’ve had far more than our fair share of issues over the years (especially those beyond our control).  I’m not complaining because such is the nature of the business and I didn’t enter it blind, but I do think that it is long past time that we caught a break.

I cover helpdesk because I actually like it and it keeps me “in the know” with what users are saying and how they are using the service.  Granted, when things are running well, helpdesk traffic is very light (we have good docs) and so it is easy, but even if it becomes more difficult with increased size I will remain in direct contact with it.

A business benefit to me covering the helpdesk is that nobody gets an auto-reply or clueless support drone.  I know I hate those when I contact support somewhere, I don’t want this service to ever suffer from that.  Support needs to know the service inside and out (and right now nobody knows it better than I do) and a person, not a script, needs to reply.

Another business benefit is one that transfers from the brick and mortar world,  people like personal attention from the owner.  There is little difference in the reasons behind why people choose a niche service and why they visit a mom and pop brick and mortar store.

If you want auto-replies and to talk to people named Bob with thick Indian accents, you go to the places so big that you are just a number .  If you want a person and even better, one with a vested interest in the business beyond just getting a paycheck, you go with a niche service.

I finally have OSX to play with, this means that I can develop on this platform now too.  This quickly allowed for the creation of a preconfigured install for OpenVPN like I did for Windows.  This makes both platforms “install and use” with our service.

VPN is just about ready for release.  Userland is finished, support docs and install pretty much done (though I will add some user contributed docs and a page on tweaks for mtu settings and fragmentation settings), automation and management is complete, all that is left is billing backend.  Should make an early Feb release.

Pricing was revamped.  I did a little due diligence  and made sure we were faster and either cheaper or close enough that it was a negligible difference for a better speed than other similar services.

The testing has been a great help.  It allowed me to work the numbers to keep the speeds very fast with a lower price point.  It’s a slim margin, but we always operate on such.  Not out to get rich (although it would be a nice change from hovering around poverty in favor of the service), but more important is great services at a good price.

I think you’re going to like our VPN.  Major service wise, this completes Cotse and provides every tool you need, next will be enhancement of existing offerings, better default quotas, more options, and possibly a little eye candy (though as a purist the thought does turn my stomach a little, but it’s what the masses want).

I really can’t believe this happened.  In case you may be unaware, the cotse.net domain name expired on the 21st.  It was immediately renewed within minutes but the damage was done.  DNS is the mistake felt round the world 

I don’t have any excuses, just the facts of what happened.  Last year I retired an old e-mail alias that I thought was not being used anywhere and was just gathering spam (love our kill aliases feature).  Unfortunately it was being used in one place, the private contact info for our registrar.  That was mistake number one.

Number two is a couple of our domains (and we have about 100 or so (not all public)) were not set for autorenew.  Cotse.net was one.  I thought it was set for autorenew.  So it came up for renewal and all warnings went to a dead alias.   I never thought about it, like the rest of us I was focused on bringing the VPN service live and then the domain expired.

And that’s what happened.  Inexcusable and I am really kicking myself.  I’d like to console myself with the fact that it has happened to others, it even happened once to Microsoft, but that is little consolation.   I cannot guarantee that there will never be another human error that causes some type of outage for something, but I can say it won’t be our main domain expiring again.  Unfortunately that doesn’t help some of you now.

The problem lies in the fact that some ISPs completely ignore the TTL and instead cache DNS for as long as they feel like it.  This means they cached the expired value and are holding onto it.  We’re completely helpless and their users will not be able to see us until they finally update (which, if your ISP is one will likely be the cause of other issues reaching sites if they change their DNS.  Something to keep in mind if you can’t reach some place and it seems like it just vanished).

While I have a lot to say about places that ignore accepted practices, this is really just our fault for letting the domain expire.   You can contact us via google or other free accounts if you still cannot access and we can provide you with a domain that will work until your DNS updates.

I’ll update more later, right now my main focus is helping users get connected.