Kaizen in infrastructure: Writing RCAs to improve system reliability and build customer trust

I shall endeavor to convince you today that your company should really regularly write Root Cause Analyses (RCAs), not just for yourselves but also as a tool to build trust with your customers. The subject of RCAs can be a bit dry, so allow me to motivate with an example of how poorly-approached RCAs can easily become a hot-button issue that will cost you customers.

As the CTO of Lottery.com, I’m responsible for overseeing roughly 50 vendor relationships. For the vast majority of these vendors, Lottery.com (LDC) is a regular customer that doesn’t use their system much differently than any other customer, and things run smoothly. They charge a credit card every month, and we get a service that fulfills some business need. Yet, for a small handful of these vendors ‘business as usual’ just can’t shake persistent issues.

Allow me to tell you of an incident that occurred recently with a vendor; let’s call them WebCorp. WebCorp offers a service that we depend on for one of our services to be up. If WebCorp’s service is up, then our service is up. If WebCorp’s service goes down, we go down. In these situations, it’s in everyone’s interest for WebCorp to be reliable. So we codify that dependency in a contract called a service level agreement (SLA). SLAs are measured quantitatively, in terms of uptime percentage.

A side note on SLAs: A high quality service measures their uptime in “nines,” as in: three nines is 99.9% uptime. That may seem like a lot, but over the course of a year 3 nines of uptime translates to nearly 9 hours of timedown, or 1.5 minutes of downtime per day. With WebCorp, Lottery.com has a five nines SLA, which translates to five minutes of downtime per year.

OK, back to the story: Lottery.com got an alert in the early afternoon around noon PDT that our service was down. Within about five minutes we had determined, conclusively, that the cause of the downtime was a service failure at WebCorp. I emailed WebCorp’s emergency helpline and, to their credit, within a few minutes they acknowledged the issue and indicated they were looking into it. About an hour later they had resolved the issue and our service was back online. Total downtime was about 64 minutes.

When a vendor has an outage it is my standard practice, once the issue is resolved, to write in inquiring about what went wrong and whether mitigation steps to prevent future outages are in place. In this case, WebCorp’s response was:

It would appear that the cache flushed right before the system tried to restart. That flush wiped the contents of a Varnish file, which caused Varnish to restart with an error. That probably doesn’t mean much to someone on your end of things. Essentially, it was a really unusual conflict of a couple of automatic jobs happening on the server, so we’re fairly sure it’s not something you’ll be able to reproduce from your end of things, intentionally or unintentionally. Hope that clarifies a bit!

While I appreciate the effort to lift the curtain a little bit on some of the technical details, this response doesn’t actually tell me how WebCorp is going to prevent the issue from happening again. And so I asked them what they planned to do to prevent future such outages. 

WebCorp’s response:

We try our very best to prevent these things from happening. In order to be better prepared for a situation like this in the future, we’ve added extra monitoring […]. Now, our emergency support team will be immediately alerted whenever any downtime happens […].

Since the issue [your service] encountered is one that we have no record of having seen (either before or since), it might be premature to alter our Varnish caching processes at this time. If the issue proves to be reproduce-able and / or widespread, then we may indeed make an adjustment to our infrastructure to correct for it. For now, though, it appears to be an isolated incident.

While you do have a 99.999% SLA with us, it is actually for a different […] service! The SLA agreement is tied to [Service2] and not [Service1]. However, you may be pleased to hear that the uptime of [Service1] has been at 99.87% over the last month!

Again, I apologize for the downtime yesterday. I hope this answers the questions you had for me and the rest of my team. If not, please feel free to reach out again so we can continue the conversation. I’m always happy to help!

Again to WebCorp’s credit, this is an undeniably polite and professionally written response. The substance, however, did little to reassure me on a technical level.

What I read in the response, substantively, is:

We’re doing our best and will add more ‘monitoring’. In fact, our support team will now actually find out when downtime occurs. But this specific issue has never happened before, so it’s not in our interest to change business practices. Oh and as a reminder, the 99.999% SLA we have for your service doesn’t technically apply here and this service been at 99.87%. Isn’t that great?

By signing and paying for a five-nines SLA, my expectation as a customer is to have as close to 99.999% uptime as possible for all services WebCorp offers. The fact that WebCorp’s response seems to indicate that they find 99.87% to be a good uptime percentage serves to dramatically reduce the trust that I have in WebCorp’s future reliability. A far more reassuring response would indicate that they take all downtime seriously, that their team is investigating ways to improve the robustness of the system to ensure no customer experiences these outages ever again and that they would reply to me in a few days when they understand exactly what went wrong in their procedures and how they’ll be improving.

In summary:

1) It is important that the vendor and customer have aligned expectations for service reliability.
2) If the vendor offers a contractual SLA, the customer’s expectation is that the vendor will make good faith best efforts to meet that SLA, and take any breaches seriously.

RCA The Right Way

By not performing and being transparent about a detailed RCA, it’s easy for a customer to lose faith in a company’s efforts to provide a highly-reliable service. The goal of the RCA is therefore twofold:

1) Document the failure and potential mitigations to improve service quality and reliability.
2) Provide a mechanism for being transparent about failures to build confidence, and trust, with customers.

A good RCA has a template roughly as follows:

Incident Start Time/Date:
Incident Received Time/Date:
Complete Incident Timeline:
Root cause(s):
Did we engage the right people at the right time?
Could we have avoided this?
Could we have resolved this incident faster?
Can we alert on this faster?
Identified issues for future prevention:

In this template are prompts for the pieces of information one needs to understand what happened, what was learned, and why it won’t happen again. There are many great examples of RCAs out there:

https://blog.github.com/2012-12-26-downtime-last-saturday/
https://medium.com/netflix-techblog/lessons-netflix-learned-from-the-aws-outage-deefe5fd0c04
http://www.information-age.com/3-lessons-learned-amazons-4-hour-outage-123464916/
https://slackhq.com/this-was-not-normal-really-230c2fd23bdc

Kaizen, the Japanese term for ‘continuous improvement’ is an ethos often cited in industry. Building technology is hard; humans are imperfect and therefore, technology often is as well. That’s expected. The only way we get past our imperfect ways is to continuously work to get better, to own up to our mistakes, learn from them, and ensure we (and our technology) don’t make the same mistake twice.

The Commandments of Good Code according to Zach

    1. Treat your code the way you want others’ code to treat you
    2. All (ok most) programming languages are simultaneously good and bad
    3. Good code is easily read and understood, in part and in whole
    4. Good code has a well thought out layout & architecture to make managing state obvious
    5. Good code doesn’t reinvent the wheel, it stands on the shoulders of giants
    6. Don’t cross the streams!

Treat your code the way you want others’ code to treat you

I’m far from the first person to write that the primary audience for your code is not the compiler/computer, but whomever next has to read the code (which could be you 6 months from now!) Any engineer can produce code that ‘works’, what distinguishes crap from those that are capable of writing maintainable code efficiently that supports a business long term is an understanding of design patterns, and the experience to know how to solve problems simply and in a clear and maintainable way. The rest of these commandments are all supporting lemmas of this thesis.

All (ok most) programming languages are simultaneously good and bad

In (almost) any programming language it is possible to write good code or bad code ergo, assuming we judge a programming language by how easy it is to write good code (it should at least be one of the top criteria, anyway), nearly any programming language can be ‘good’ or ‘bad’ depending on how it is used (or abused).

An example of a language that by many is considered ‘clean’ and readable is Python. Many organizations will enforce a universal coding standard (i.e. PEP8), the language itself enforces some level of white space discipline and the built in APIs are plentiful and fairly consistent. That said, it’s possible to create unspeakable monsters. For example, one can define a class and define/redefine/undefine any and every method on that class at runtime. This naturally leads to at best an inconsistent API and at worse an impossible to debug monster. Yes, one might naively think, but nobody does that! Unfortunately that is untrue, and it doesn’t take long browsing pypi before you run into substantial (and popular!) libraries that (ab)use monkeypatching extensively as the core of their APIs. I recently used a networking library (xmppy) whose entire API changes depending on the network state of an object. Imagine calling ‘client.connect()’ and getting a MethodDoesNotExist error instead of HostNotFound or NetworkUnavailable.

As an example of a language that by many is considered ‘dirty’ but can be quite pleasant is Perl. Now, I know the previous sentence will cause a lot of controversy, so rather than fend off the pitchforks myself, I’ll refer you to Dave Cross, a proper Perl expert who very eloquently discusses this very topic.

Good code is easily read and understood, in part and in whole

Good code is easily read and understood, in part and in whole, by others as well as the author in the future (Trying to avoid the ‘Did I really write that?’ syndrome). By in part I mean that if I open up some module or function in the code I should be able to understand what it does without having to also read the entire rest of the codebase. Code that constantly references minute details that affect behavior from other (seemingly irrelevant) portions of the codebase is like reading a book where you have to reference the footnotes or an appendix at the end of every sentence. You’d never get through the first page! Some other thoughts on ‘local’ readability:

    • Well encapsulated code tends to be more readable, separating concerns at every level.
    • Names matter. Activate system 2 and put some actual thought into names, the few extra seconds will pay dividends,
    • Cleverness is the enemy. When using fancy syntaxes such as list comprehensions or ternary operators be careful to use them in a way that makes your code more readable, not just shorter.
    • Consistency in style, both in terms of how you place braces but also in terms of operations improves readability greatly.
    • Separation of concerns. A given project manages an innumerable number of locally important assumptions at various points in the codebase. Expose each part of the codebase to as few of those concerns as possible. Say you had some kind of people management system where a person object may sometimes have a null last name. To somebody writing code in a page that displays person objects, that could be really awkward! And unless you maintain a handbook of ‘Awkward and non obvious assumptions our codebase has’ (I know I don’t) your display page programmer is not going to know last names can be null and is probably going to write code with a null pointer exception in it when the last name-being null case shows up. Instead handle these cases with well thought out APIs that different pieces of your codebase use to interact with each other.

Good code has a well thought out layout & architecture to make managing state obvious

Sublemma: State is the enemy. It is the single most complex part of any application and needs to be dealt with very intentionally. Common problems include database inconsistencies, partial UI updates where new data isn’t reflected everywhere, out of order operations, or just mind numbingly complex code with if statements and branches everywhere leading to difficult to read and even harder to maintain code. Putting state on a pedestal and being extremely consistent and deliberate in how state is accessed and modified dramatically simplifies your codebase. Some languages, Haskell for example, enforce this at a programmatic level. You’d be amazed how much the clarity if your codebase can improve if you have libraries of pure functions that access no external state, and then a small surface area of stateful code which references the outside pure functionality.

Good code doesn’t reinvent the wheel, it stands on the shoulders of giants

In 2015 we have a wealth of tools available to solve common problems — depend on them as much as possible so you can focus on solving the interesting part of your application. Think ahead of time of somebody has solved some portion of the problem you’re trying to solve. Is there something on the interwebs/github (with an appropriate license) that you can reuse? Yes, expect to make heavy modifications, but more often than not this is a time saver. Things you should not be reinventing in 2015 in your project (unless these ARE your project)

Databases.

I don’t care what your requirements are, it exists. Figure out which of CAP you need for your project, then chose the database with the right properties. Database doesn’t just mean relational databases (e.g. MySQL) anymore, you can chose from any one of a huge number of data storage models:

    • Key Value Stores e.g. Redis, Memcache
    • “No-SQL” e.g. MongoDB, Cassandra/
    • Hosted DBs: AWS RDS / DynamoDB / AppEngine Datastore
    • Map Reduce engines: Amazon EMR / Hadoop (Hive/Pig) / Google Big Query
    • Even less traditional: Erlang’s Mnesia, iOS’s Core Data

Data abstraction layers

You should, in most circumstances, not be writing raw queries to whatever database you happen to chose to use. There exists a library to sit in between the DB and your application code, separating the concerns of managing concurrent database sessions and details of the schema from your main code. At the very least you should never have raw queries or SQL inline in the middle of your application code, please wrap it in a function and centralize all the functions in a file called ‘queries.py’ or something else equally obvious. A line like users = load_users() is infinitely easier to read than users = db.query(‘SELECT username, foo, bar from users LIMIT 10 ORDER BY ID’) etc. Centralization also makes it much easier to have consistent style in your queries, and limits the amount of places to go to change the queries should the schema change.

Other

    • Between S3, EBS, HDFS, Dropbox, Google Drive, etc. you really shouldn’t spend much effort or mental energy on storage now a days.
    • Take your pick of queueing services provider — ZeroMQ/RabbitMQ/Amazon SQS

Don’t cross the streams!

There are many good models for programming design, pub/sub, actors, MVC etc. Choose whichever you like best, and stick to it. Different kinds of logic dealing with different kinds of data should be physically isolated in the codebase (again, this separation of concerns concept and reducing cognitive load on the future-reader). The code which updates your UI should be physically distinct from the code that calculates what goes into the UI.

Conclusion

This is by no means an exhaustive or the perfect list of Good Coding Commandments. That said, if every codebase I ever had to pickup in the future followed even half of the concepts in this list I will have many fewer gray hairs and add an extra 5 years on the end of my life and the world would be a better place.

Credit to Scott Kyle (appden) for assistance reviewing the material in this post. Have you gotten Current For Mac yet?

The Product Market Fit Flow Chart (or PMFFC for short)

The goal of every startup, or any company producing a product that has some kind of ‘customer’, early on should be to find that magical ‘product market fit’ (herein PMF because I’m lazy). According to Wikipedia:

Marc Andreessen was the first person that used the term: “Product/market fit means being in a good market with a product that can satisfy that market.”

I was recently being recruited by a company, but turned them down because of what I saw was a big business mistake. They had PMF, but they weren’t doing anything with it.

Allow me to introduce what I believe should be on a laminated pamphlet and given to every entrepreneur in the valley:

PMF

The bit that was missing in the company I was talking to was the Run With It stage. By that I mean make your customers happy and as quickly as possible get as many as possible! This particular founder told me about the customers he already had, and about his 8-9 month roadmap to hire engineers and build product. I asked how many sales and support people he would hire in that time, he said zero.

By the roadmap he was suggesting in 2016 they would be taking in maybe ~$10k monthly, have an engineering team of ~10, a sales team of 1 (the founder) and an annual burn of >$1MM. An alternative reality, would be by 2016 to have an engineering team of 5, a sales and support team of 3-5, taking in $100K monthly and be on the verge of break even and able to raise a kickass series A.

I’m a technical person by training, and 5 years ago I’d probably be making the same mistake as this founder.  It’s really unnatural to think about sales, and it’s easy to think of them as unskilled workers you can pickup by the dozen at a moments notice.  In reality, this couldn’t be further from the truth.  Sales is hard, very hard, and you need to be working on it as early as possible.  I consider it one of the most important lessons I’ve learned as an entrepreneur to never undervalue or under-prioritize your sales and distribution strategy.  Once you’ve achieved PMF, or are even within long distance sonar range scanners of PMF, you should be thinking really hard about how to sell the bananas out of what you’re building.

Vietnam Bike Tour: Hanoi to Ho Chi Minh City (Day 9)

DAY: 09Screenshot from 2014-10-15 08:43:39
Distance: 308KM
Origin: Da Lat
Endpoint: SAIGON!!!!!
Pho Consumed: 1 each for lunch
Today’s Author: Zach

Screenshot from 2014-10-15 08:43:31

Well, today’s the day! We estimate its roughly 290-300KM to Saigon, which should be doable in one day, we think. Both of our bottom end’s have been pretty painful lately due to the number of hours riding, so we’ve taken to starting the morning with a dose of advil to ease the discomfort while riding which of course requires food.  Last time we were in this predicament we had some of our “emergency rations.”  This time, though, the hotel/guesthouse we’re staying at offered us a complimentary breakfast — they made fresh eggs, had fresh bread and lots of nicely sliced fruit. I tried Marmite for the first time — good god the stuff is awful. Absoultely disgusting. Worse than China’s “stinky tofu”.

As we were packing up the lady at the hotel (whose English was impeccable — a definitely surprise for us) asked where we were headed. I told her our goal was to make it to Ho Chi Minh city today, and she said “no no, too far, need 2 days.” Uh oh. Not a good sign. Hopefully we can prove her wrong, as neither of us was really looking forward to spending another night in a middle-of-nowhere-sketchy-motel.

We got on the road around 9:45AM. We only rode about 200km the day before so were not urgently pressed for gas, so we simply got started It took less than 10 minutes to leave town and shortly after that we were back in the wilderness, driving down the steep mountain that Da Lat is based on. The roads were in pretty reasonable shape on the way down — nothing too remarkable.

We got to the bottom of the mountain and ended up on this really very nice, properly divided 4 lane highway. It felt rather out of place given everything else we’de seen in the country so far. Even stranger, it soon divided into a “cars only” and “motorbikes only” roads (two completely separate roads). Normally we’re actually going faster than the cars on the road (at a whopping 90KM/h), so it might’ve made practical sense to go on the car highway, but given that it’s the last day we didn’t want to get into any unecessary trouble, so we ended up on the bike lane.

Shortly later, about 20KM into the day, we stopped for gas. I was looking at the map at the gas station and realized that the other end of the “car boulevard” was actually the Da Lat airport. It all makes sense now! The wealthy folk land at the airport and get a basically traffic-free ride straight into one of the nicest towns in the country.

Thankfully neither bike was short on oil and we quickly got back on the road. The highways were pretty typical fair — with a mix of rural open highway without much traffic, and towns at regular intervals with tons of traffic (read: dodging semitrucks coming straight for you and forcing us into the dirt, which we got very good at riding in). South Vietnam definitely feels denser than the north — the distance between the towns felt smaller and each town felt larger, though the towns were still pretty typical fair — mechanics, Pho restaurants, tiny stores selling the same soda/tea and unlabeled shantybuildings.

At about 140KM, 1PM we stopped for lunch at — you guessed it, a random tiny Pho place. This place was covered in dogs. At least 10 dogs, most of which looked to be puppies were running around. None of them were particularly clean looking or well groomed. Aside from moan about the heat and how soar our butts were we debated whether or not these dogs were pets, or food. We didn’t have the heart to use the translator to ask, so it’ll remain a mystery for now. By 1:30 we were back on the road with what Google said was 148KM to go.

 

Somewhere in this last stint we came upon stopped traffic. The road was completely blocked. Half of the road was stopped by a giant flatbed truck with big concrete cylendars on it (presumably some kind of municiple plumbing project) and the other half was a truck whose mirror couldn’t clear the concrete tubes on the flatbed. For reasons we still don’t understand, the truck couldn’t back up, and turn the wheel left and move over 6 inches to clear the flatbed. Instead he simply stood there and all of traffic waited and watched the flatbed crane move each of the cylendars off the flatbed and onto the ground so the truck could pass. What insanity!

At about 50KM outside the city we reached a fork in the road. Things were starting to feel a bit denser and it felt as if we were on the outskirts of the city, which was really an incredible feeling. The map took us in a way we didn’t expect, which turns out to be really nice. For about 25KM we had nothing but straight, 4 line highway with proper dividers (so nobody turns into you) and were able to do a solid 100KM/h without the constant thread of kamikaze-motorbiker or passing-in-oncoming-traffic-18- wheeler-going-to-turn-you-into-pancake.

After the nice highway we knew we were in Saigon. How? Because there was traffic. Not like many motorbikes in my way traffic, like bumper to bumper cars going slowly on a highway traffic. First time we’de seen that on the journey! It was slow going, and we were forced to breath in a lot of diesel fumes, but we eventually made it to the hotel in Ho Chi Minh City. WAHOO!

We dropped all of our stuff in the room and headed back out on the bikes for the last time. We drove into the heart of the city and got to see upclose how different Saigon is from the rest of Vietnam. My first reaction was “this isn’t Vietnam, there’s Fancy Stuff here!”. Mercedes, BMWs, Rolls Royce cars, Starbucks! Oh my.

We had some difficulty finding where to drop off the bikes as Google’s pin for the address was (fairly typically for this part of the world) about 8 blocks off. We did however find it eventually! They did a quick inspection of the bikes and discovered Felix’s broken rear suspension when trying to move the bike by hand. They picked up the rear of the bike and bits of the shock absorber literally fell out. That’s not supposed to happen!

A quick call to Mr. Hung and we got the final bill. Cost for the new wheel and the bike’s suspension: $96. Not too bad all and all. We settled everything, got our deposits back and sat in a taxi back to the hotel. Oh what a feeling to be on four wheels and have air conditioning again! We took a quick dip in the pool to begin the de-stinkification process. Followed by showers (long ones, with lots of soap and scrubbing). We got a few restaurant recommendations from friend’s and headed over to “District one” via taxi. The first restaurant we wanted to go to (Saffron) was actually so full and busy that they told us “sold out for the night!” That’s a new thing.

We tried the sister restaurant (Italian) which only had a 10 minute wait. Still unusual, but we didn’t mind waiting 10 minutes. I’ll spare you most of the dining details, suffice is to say the food was delicious and up there as one of the best Italian restaurants we’ve ever been to. Including a giant 2 foot wheel of parmasean that had its top layer scraped off and then Felix’s pasta literally coated in the fresh stuff. Total bill was $35 a person, including appetizers and wine. Expensive for Vietnam, but very cheap for the quality anywhere else in the world!

Made it back to the hotel and we’re both promptly ready to pass out after a long Journey!

Stay tuned for the epilogue, including some data about our trip and retrospectives!

2014-05-21 20.50.55

Vietnam Bike Tour: Hanoi to Ho Chi Minh City (Day 8)

DAY: 08Screenshot from 2014-10-15 07:40:46
Origin: Nha Trang
Destination: Dalat
Distance: 140km
Pho Consumed: 2
Today’s Author: Felix
We woke early for our diving trip, heading downstairs to the dive center around 7am (convenient, right?).  Since the bus wasn’t scheduled to depart until 7:30, we headed across the street to a restaurant with Russian signage, which seemed to be the only place open on our street. We had a decent western-style breakfast, but unfortunately had one of our only poor service experiences in Vietnam; the food took nearly 30 minutes to appear, even though we had simply ordered some coffee and toast.  After trying (and apparently failing) to impress upon the staff our need for expediency, we finally made it back to the dive center around 7:45.
Awaiting us were our companions for the day, a handful of French tourists and Natalie, a Malaysian-Canadian girl on the second day of a dive certification class.  Fortunately everyone seemed cool with our tardiness, and we all piled into a minibus and headed for the dock.
Arriving at the dock, we hopped on our boat and met the rest of the dive crew as we headed away from shore.  Our divemaster for the day, Nguyen, was a chill dude who spoke pretty good English, and we felt that we were in good hands.  During the 40 minute ride to the dive site, we got a refresher on some basic scuba skills since Felix hadn’t dived in a number of years and Zach isn’t certified.
Arriving just off the coast of a small island, we were pleased to see that the water was almost perfectly clear.  After getting geared up, we followed Nguyen and hopped into the water.  We spent the first 10 minutes or so reviewing our basic scuba skills (hand signals, mask clearing, buoyancy control, etc.) and headed off after Nguyen to explore the reefs.  We were also accompanied by a cameraman who did a bang-up job of capturing our adventures.
While perhaps not the most amazing diving in the world, the reefs were teeming with life.  Although we stayed in relativey shallow water (~8-10m), we saw a host of incredibly colorful smaller fish, as well as a plethora of anenomes and coral.  We also played with a giant jellyfish (!), which was apparently safe to touch as long as you stayed away from the tentacles on the bottom.
After about 50 minutes, we headed back to the boat and enjoyed a light snack of mangoes and baguettes.  Felix also sampled a proffered Vietnamese cigarette – apparently the “White Horse” brand is produced locally in Nha Trang.  We then geared back up and headed down for another dive.
On our second dive, we went a bit deeper – perhaps 12 or 15 meters.  For the most part, the marine life we saw was pretty similar to our first dive. We did briefly glimpse a school of larger fish about a dozen meters away, but they swam off fairly quickly.
Returning to the boat once more, we hung out for a bit as we waited for the rest of our group to return before we headed back towards the city.
Back on land once more, we piled back into our minibus and headed over to a small cafe for lunch, which was included in the price of our trip for the day.  We enjoyed some pretty tasty noodles accompanied by mystery meat, but we trusted our French tour leader and chowed down (apparently the cafe is owned by his Vietnamese wife).  The noodles were followed by crepes slathered in mango sauce for dessert, which were delicious if a bit too saucy.  We also chatted with Natalie about her traveling adventures throughout Asia, which was a nice change from our usual mealtime discussions of tech, business, and motorcycle-related soreness.
We headed back to the dive center / guesthouse, took quick showers, and packed up our bikes.  After settling our bill and getting directions to a Honda mechanic, we set off.  Unfortunately, it turns out that our bikes (XR150s) weren’t sold by Honda in Vietnam, and the dealership was thus very confused about our request for maintenance services.  We got directions to another garage in the more “Vietnamese” area of the city and headed off in that direction, but were unable to sufficiently explain our issue to the mechanic (Felix’s headlight was exceptionally dim).  Giving up, we gassed up and left the city around 4.
Heading towards Dalat, we were greeted by a new road in relatively good condition, as well as some very threatening stormclouds in the distance.  However, we got incredibly lucky and had some of the best riding of the trip; there was almost no traffic, the roads were both excitingly twisty and in pretty good shape, the scenery was incredible, and the temperatures dropped rapidly to a pleasantly cool level.
We passed a number of natural waterfalls, as the storm had just passed and there was plenty of water on the ground and in the hills.  Rapidly gaining elevation, we were treated to some incredible vistas over the mountains as the sun was beginning to set.
After a couple of hours of really outstanding riding, dark began to fall and we started to get a bit chilly.  On our way into the city of Dalat, we passed a dozen or so kilometers of lit up greenhouses, which made for some very pretty scenery.  We also began to notice large, American-style McMansion type homes; Dalat is clearly a city favored by wealthier Vietnamese.  This became more apparent as we entered the city proper; compared to much of the rest of Vietnam, the city is relatively well-kept and well-developed.  Near the center of town is a charming small lake, and nearby is a massive golf club.
We arrived at a small hotel recommended by Lonely Planet, the Dreams Hotel, and were pleasantly surprised to find that the elderly couple who run the place were both very friendly and spoke decent English.  We were also shocked to hear their adorable grandchildren running around the lobby speaking perfect English without a hint of accents whatsoever.
After checking in to the room and availing ourselves of the modern shower facilities, we headed down the street to grab some dinner.  We quickly stumbled upon a pizzeria that seemed promising and enjoyed some surprisingly decent pies (“Four Cheese,” “Mexican,” and Margherita).
Following dinner, we decided to wander around a bit.  After passing a number of bustling restaurants, bars, and shops, we happened upon a fancy-looking bakery.  As we hadn’t gotten Zach any cake for his birthday nor had dessert yet, we popped in and, after a quick glance at the prices, started piling a tray with refined carbohydrates (plus two party hats, because why not?).  Total bill for our sugary indulgences?  About $5.  What an awesome country!
Heading back to the hotel, we dove into our haul.  Zach put his friend Leona on video chat so she could see us in our party hats stuffing our faces with cake and sweets in the middle of Vietnam.  After polishing off most of our dessert, we headed to bed after what was clearly one of the best days of the trip.

Vietnam Bike Tour: Hanoi to Ho Chi Minh City (Day 7)

Day: 07Screenshot from 2014-10-15 07:11:08
Distance: 287km
Origin: Binh Duong
Destination: Nha Trang
Pho Consumed: 1
Today’s Author: Felix

Screenshot from 2014-10-15 07:11:02

Somehow missing our alarm, we still managed to be up by 8.  In the interest of expediency, we had the rest of our emergency-ration faux-moon-pies for breakfast.  Leaving the hotel at 9:55, we gassed up and did a quick maintenance check on the bikes before we hit the road.  The road along this stretch was in pretty good condition, and we passed a few gigantic golden buddhas and temples as well as some very picturesque beaches.  We covered about 130km before stopping for lunch at a small shop in a seaside town and grabbing a bowl of Pho.

Heading back onto the road, we passed more beaches and reached Nha Trang around 4:15 in the afternoon.  On our way into the city, we passed through the non-touristy part of town, went past a gigantic temple of some sort and over a nice bridge surrounded by colorful fishing boats.
2014-05-19 15.56.54
Video 1:
Video 2:
Nha Trang is much more developed than anywhere else we’ve been in Vietnam, with high-rise luxury hotels lining the pristine beach.  The streets are crowded with Westerners frequenting the many hotels, bars, restaurants, and so forth that are all designed to cater to them.  One thing we were surprised by was the prevalence of Russian signage – something we haven’t seen anywhere else in the country.  We later learned that the city is a major vacation destination for Russian tourists, with as many as 10 direct flights from Moscow arriving daily.
We checked in at Angel Dive Center and Guest House and made arrangements for our scuba outing the following day.  After cleaning ourselves up a bit, we headed down the block to a high end beach bar and restaurant for some birthday drinks for Zach.  The prices were ludicrous, especially for Vietnam, but the setting was incredible and the drinks were delicious.
Heading inland a couple of blocks, we had dinner at a nice Vietnamese place called Lanterns which was very highly rated on Tripadvisor and consequently packed with tourists.  To one side of us were a pair of very creepy Russian gentlemen (one older and one younger), and to the other was a large group of Korean sailors on shore leave.  The food was pretty good, and we hit up an extremely legit gelato shop for dessert – doing the tourist thing certainly has its advantages!

 

Vietnam Bike Tour: Hanoi to Ho Chi Minh City (Day 6)

DAY: 06Screenshot from 2014-08-08 20:41:22
Distance: 370KM (-50 from goal)
Origin:Hue
Endpoint: Middle of nowhere 50km shy of Nha Trang
Pho Consumed: 1 each for dinner
Today’s Author: Zach

Screenshot from 2014-08-08 20:41:10

We targeted more than 400KM today, including a 3-4 hour stopover in a city called Hoi An.  That roughly translates to 8 hours of riding, plus 3 hours of tourism, would mean we’de need to take advantage of every minute of daylight that we had.  As such, the alarms went off promptly at 5:30AM.  And then again at 5:40AM.  And again at 5:50AM.  By 6 Felix was in the shower and my eyes were partially open.

We finally stumbled out of the “Google Hotel” for the last time around 7AM, ready to hit the road bright and early.  As we road south towards Hoi An we quickly found out that our headaches with Rt. 1 were not yet over, and in fact that road was a miserable bus-overtaking-truck-in-a-2-lane-highway-move-or-become-pancake the whole way.  Every now and then we would ride through a town where it would open into a 4 lane highway with a center barrier which was wonderful.  Until the breaks in the barrier every 200 meters or so where motorbikes would pull U-turns without even looking and nearly run into whomever happened to be in the fast lane on the highway.  Maybe this system of lawlessness works in Hanoi where everybody is doing 20kmph the whole time, but when you have cars and motorcycles doing 100+ on a 4 lane highway simply darting into the road becomes a nonstop series of heart attacks.

To add to that, we had a bit of mechanical trouble with the chain on my bike.  Pulling over to the side of the road, taking out the toolkit and working those good-ol mechanic muscles did the trick.  Not without a good bit of extra sweat though!

IMAG1261

I’ve also had an interesting thought.  There are so many motorbikes, each following an individual routing algorithm, that it actually visually looks like a fluid flow.  I witnessed the stream of bikes moving about 60kmph on a highway and watched a bus drive perpendicularly through the road.  The bikes slowly bent around the front of the bus — continuing to pass of course, as the bus crept slowly into the road.  As soon as the rear of the bus was onto the highway the bikes were now going around the bus on both sides.  As more and more of the bus moved through the highway the flow of bikes slowly closed off on the front, and continued opening up behind the bus.  Literally like the bus was moving through a river of motorbikes.

So, enough whining, time for some good stuff.  Felix claims that just outside the next major city, Da Nang, Jeremy Clarkson and Top Gear went up and over a mountain and Clarkson labeled it one of the best drives in the world, and he would know!   Now-a-days they’ve built a tunnel through the mountain, but with that kind of recommendation, we decided to go up and over.  We were not disappointed!  The views were spectacular, and we even found a heard of goats on the way up!

At some point on the mountain we realized that we didn’t have any pictures of us actually riding the bikes, so we stopped to take some action shots.  Over 300 of them, as it turns out.

anim2small anim1small

 

We mostly just sailed straight through Da Nang, but with one exception.  The main highway through town, very abruptly, was completely closed.  And the river of motorbikes didn’t seem to mind at all — they just turned left and right.  I made the instinctive decision to simply follow the river.  We ended up being part of a 1KM long snake of motorbikes winding through these tiny 1 meter wide alleys.  It was the most ridiculous thing.  These alleys are probably normally just people going about their business on foot, but today was literally host to a constant stream, with zero intreruption, of bikes going way too fast with literally no space inbetween them.  We eventually wound our way out of the alleys and back onto the highway.

We sailed straight past our destination of Hoi An by about 8KM.  Our first legitimate wrong turn in ages.  We turned around and found a tiny hidden offramp that was literally a steep dirt downhill.  No wonder we missed it the first time!  We followed this awfully skimpy excuse for a road for about 5KM east to go from the main highway to Hoi An.  This road could more accurately be described as an alleyway than a road.  Whilst on the almost-a-road-but-not-quite a motorbike pulled up next to me with a girl on the back who started yelling at me asking about what hotel we were going to etc.  Skeptical of the free advice, and also mostly trying not to fall off this tiny road or run into a wall, I managed to ignore/shoo off the shouting girl.

We finally got to Ancient City of Hoi An and promptly got yelled at by some security guards that we couldn’t ride our motorcycles in.  Instead they pointed us to this fat dude with half a shirt who would “park” our bikes for the day.  Without much choice, we handed the guy the equivalent of $2 and parked.  It was near 11AM and neither of us had eaten, we were also extremely hot, sweaty and dehydrated (read: irritable).  So, we sat down at the first or second restaurant we found, and did the usual TripAdvisor double check, 4.5 stars, excellent!

2014-05-18 11.34.58

2014-05-18 11.35.02

2014-05-18 11.35.05

2014-05-18 11.52.582014-05-18 11.42.46

 

 

 

 

 

 

 

After a delicious but a bit excessive lunch we set out to look at some old stuff.  We looked at some old stuff.  Here are pictures of some of the old stuff.  It was old.

Around 2PM we made our way back to the bikes to find that the Fat Man had not stolen all our stuff, yay!  Our helmets were nowhere to be found though.  We tried knocking on the door of what we thought was the Fat Man’s shop but didn’t get an answer.  After a few minutes a nice lady from the shop across the street called to us and pointed at her head.  We pointed at our heads.  She then pointed to behind some manikins in her store where, voila, there lay our helmets.  Onwards!

We filled up for gas outside of town and set fourth towards our destination: the biggest city within spitting distance of Nha Trang, our destination for day 7 and the next major stop in our journey.

The roads were, unfortunately, none better than this morning.  Highway 1 is an absolute nightmare the whole way.  We road for a solid 4 hours and covered less than 200KM before it started to get dark out. Averaging less than 50kmph is really frustrating, constant stop and go.  Felix described it as a “constant battle” and it was.  We pulled over at dusk and discussed night riding again.  Our past experience was pretty obscene and we were both keen to avoid a repeat.  However, the road was in much better condition, there was less traffic and I posited that perhaps it might be acceptable.  We decided to give it a try for a short while and see how it went, and pull over if it felt dicey at the first town we could.

We made it about 25KM, or about half an hour, before it started to feel stupid and unsafe.  We stopped at this giant 4 story building (very rare and unusual for these sorts of towns) that had a big sign outside that said “HOTEL” (amongst other unintelligible things).  We were greated by a giant family of about 12.  A whole bunch of kids, a tiny dog, what we assume is mom and dad as well as grandma.  Clearly a family business.  We negotiated a price by typing numbers into my phone ($10 for the night) and proceeded to move our things upstairs.  The room had AC and a fan and the beds weren’t made of lead, so to me that was success.  We took a quick rest before going out on our usual “find a sign that says Pho” journey for food.

About 1 minute into our “Find the Pho” journey we found the Pho.  We ordered the Pho (by saying Pho and holding up 2 fingers for 2 people).  They gave us Pho.  The Pho had meat in it, but it wasn’t the usual Pho kind of meat.  It was some kind of fake Pho meat, so we wondered.  So we didn’t eat the fake Pho Meat, just the Pho.  It was good Pho.  We paid $1.50 each for Pho.  Pho-inished.

 

Vietnam Bike Tour: Hanoi to Ho Chi Minh City (Day 5)

DAY: 05 (tourism in Hue day)
Distance: 0KM
Origin:Hue
Endpoint: Hue
Pho Consumed: 0
Today’s Author: Zach
Screenshot from 2014-07-01 10:27:59
Woke up early around 9:00 AM with the intention of doing some morning tourism and mid-afternoon/evening ride to the next town.  As usual though somehow it takes us a bit longer to get everything ready to go and out the door.
2014-05-17 10.35.52
We ended up downstairs talking to the front desk about tourism etc. sometime around 11.  (I said we weren’t so good at waking up…).  We had a checklist of things we needed to do though:
  • Horn stopped working on my bike during the rain yesterday, need to get that fixed
  • Both bikes need oil changes / chain lube
  • Pick up laundry from the night before
  • Find some way to do tourism in Hue
  • Find a place to store our bags while doing tourism.

We managed to figure most of these things out and were on the bikes heading towards a “Honda repair” place somewhere around 11:45.  Predictably, the place the hotel guided us didn’t exactly exist, but there are enough scooters in this country that tiny corner mechanics are fairly easy to find even without a map.  We pulled up to this tiny hole in the wall and attempted to explain the maintenance/repair work we wanted done.  I think we successfully communicated most of it and he started working.  I then called the actual owner of the bikes to let him know what we were doing to them, and to confirm with the mechanic that everything was in order. 2014-05-17 11.34.182014-05-17 11.34.09

 

 

 

 

 

 

20 minutes later, a swapped fuse and two quarts of oil, $13 and both bikes were good to go!  Funny how these bikes have a grand total of one fuse in the whole system!  My car, from 1985, has at least 50, and this motorcycle from 2013 has one!  Sorry, this blog is about our adventures in Vietnam, not pondering on the mechanics of cars/bikes. We got back to the hotel and decided we were overdue for some local style ‘Foreign’ food.  The hostel was able to graciously provide (these guys have proven that they will literally do *anything* to take our money).  We, with a bit of trepidation, ordered a pizza and a burger.  Both because we had been eating some pretty sketchy Vietnamese food for the past week and also because we were curious how well they did American cuisine.  I’ll spare you too many details:  they get a solid B.  Burger and pizza were both well within the range of acceptable, though of course far from spectacular. 2014-05-17 12.05.582014-05-17 11.57.29           We then got in our private car with driver AND tour guide, for just us.  (<mechanic> The car was a rebadged Chevy Equinox, no idea what the local name was.  Either way, very fancy for the local culture, and had AC!</mechanic>) We set off for our first destination: the Citadel and Forbidden City.    You can read about these locations online at Wikipedia if you want to know all the actual history, some of our photos are below as well.  Our personal experience was rather straightforward.  The most notable bits:

  • Our tour guide spoke four languages: English (ish), French, Italian and of course Vietnamese.
  • He is afraid of China (politically)
  • He’s a bit bitter at America for blowing up about 80% of the forbidden city in the Vietnam War (which they call somewhat unsurprisingly call the ‘American War’).
  • Even sacred destinations like the Tomb of an King are covered with these hyper tourist-focused shops selling trinkets and soda.  It really ruins the atmosphere in my opinion.  Maybe one shop at the entrance for cold water, but beyond that it’s a bit tacky.

Post touristing we decided that night riding again wasn’t in the cards, and we would instead wake up early the following morning and do some early morning riding.  So, we checked back into the hotel and moved all of our gear back up into the hotel room. After what seemed like an eterinty Felix finally made up his mind about where we would eat dinner.  Apparently cross referencing Lonely Planet, Trip Advisor and personal recommendations from friend’s is a tedious process!  Sounds good to me 😉

The place turned out to be pretty fantastic.  The menu was massive and neither of us could make up our mind as to what to eat.  Thankfully the restaurant foresaw this problem and has a variety of “Menu Europeen” style items on the back — price fixed menus.  We chose a pair of the price fix menus ($15 each for 4 course meals) and relaxed.  The food arrived bit by bit and was all delicious.  I would describe it as Vietnamese style but with European influences.  Sadly our very refined waitress did not speak any French, though we found out on the way out that the proprietor was himself French. Landed back at the hotel around 9PM on a Saturday.  We’re heading to bed around 10PM — the lamest travelers you ever saw!  5:30AM wakeup, alarm set!

Vietnam Bike Tour: Hanoi to Ho Chi Minh City (Day 4)

DAY: 04Screenshot from 2014-06-15 01:05:26
Distance: ~300km
Endpoint: Huế
Pho Consumed: 0
Today’s Author: Felix
Screenshot from 2014-06-15 01:05:12

Despite our intention of getting an early start, we managed to ignore our alarm and didn’t get out of bed until around 10.  After grabbing a quick bite (seafood soup, rice, more egg, and mystery meat) and loading up the bikes, we hit the road around 11.  Departing from the humble village-without-a-name, we headed into some excellent riding roads.  Although the temperature was scorching, we encountered little traffic and had a pretty enjoyable ride.  However, we did see our first Westerners in a couple of days – first a pair heading north, one of whom had a GoPro on his helmet, then a few shirtless gentlemen on scooters in the twistys.  We stopped after about 30 minutes to snap some photos, then continued on.

The end of the mountain roads deposited us in a deep valley at the north end of the Phong Nha park area, with a nice wide highway.  Around 12:15, we stopped for fuel and to check our maps, then headed into the park towards Paradise Cave.  Despite a distinct lack of signage or instructions for tourists, we made our way 20km into the park and found ourselves at the entrance to a tourist site around 12:45.  We parked our bikes (5,000 VND) and grabbed some cold drinks, then headed off towards the cave after buying entrance tickets (120,000 VND).  Presented with the option of hiking towards the cave mouth or taking a small electric cart, we decided to walk (hey, we could use the exercise, right?).
A 15 minute or so walk on along a paved path brought us to the base of some stairs, where we began our ascent towards the cave entrance.  Along the way, we encountered a friendly gent from SF who had hired a guide; she was quite helpful in directing us towards the cave entrance.  Reaching the top of the stairs after another 15 minutes or so, we headed into the cave.
Phong Nha is noted for its extensive cave systems, which extend for some 70km underground; only 20km or so of these have been fully mapped.  Visitors to Thiên Đường (aka Paradise) Cave is one of the largest caves in the world, extending for some 31km underground, with the larger chambers over 100m in height.  Tourists are permitted to explore the first kilometer or so on their own, and up to 7km if accompanied by a guide.  Being neither guided nor experienced spelunkers, and wanting to make Hue by nightfall, we spent about 45 minutes exploring the blissfully cool caves before heading back out, stopping briefly for a cold drink.  In the interest of time we also opted to skip Phong Nha cave, which is supposed to be spectacular, as it contains a large underground river.
Heading back onto the highway around 3 (after another cold drink), we made fairly good progress until we hit a massive thunderstorm.  Stopping briefly to acquire/put on rain gear, we quickly became soaked anywhere we weren’t covered by waterproof clothing.
 Although we reduced our speed in the interest of safety, we made it through the worst of the storm after about an hour.  We stopped at a gas station around 5:40 to check our maps and get fuel.  With the light failing rapidly, we began to encounter some fairly dangerous riding conditions.
First, the Vietnamese seem to have in indifferent attitude towards the use of headlights, even after dark, causing some fairly obvious problems.  This is exacerbated by the fact that motorbikes can (and often do) drive in the shoulder of highways, leading larger vehicles to think it’s perfectly safe to move into the oncoming lane to pass (two wheelers are expected to get onto the shoulder and out of the way).  Third, most of Vietnam’s highways are unlit at night, which only compounds the headlight issue.  Finally, construction zones appear to be ubiquitous, frequently necessitating rapid maneuvering and going quite slowly.
We also had to stop at train crossings twice, which saw just about everyone (trucks, vans, bikes, etc.) jockeying for position – including moving into the opposing lane on both sides of the crossing!  Fortunately our bikes had enough power to get us away from the worst traffic snarls.  Between the traffic, the train crossings, the poor weather, and the lack of light, we reached our hotel in Hue around 7:45 (including a harrowing ride across a 4m wide bridge absolutely PACKED with scooters over the river in the city).
2014-05-16 21.23.00
Hue is the former imperial capital of Vietnam.  The city is divided in half by the Pearl river, with the older part of the city (and the famous Citadel) to the the northwest of the river, and the newer portion of the city to the southeast.
When we saw the “Google Hotel” listed in our guidebooks, we knew we couldn’t pass it up – although they didn’t seem to understand Zach’s request for an employee discount.  Fortunately, the hotel was clean, offered us free beer, and took our dirty laundry.  After a quick rinse and change, we had a beer in the lobby and stopped by a large electronics shop.  After a lot of gesticulating, the staff were able to help us finally get our cell phones’ internet speeds back up.
IMAG1171
We then headed to dinner at a tiny shop recommended by one of Felix’s Wharton classmates, Bánh Bèo Cung An Địnhh, where we enjoyed some traditional Hue food (mostly different sorts of rice cakes with shrimp, meat, and other bits, served with a slightly spicy and very salty/oily fish sauce).  The place was literally a hole in the wall – 20m down an alley with no visible signage – but the food was delicious.
We then headed back to our hotel, stopping at a shopping mall and getting soft-serve at an American-style fast food joint.  On the way, we also saw a few Vietnamese guys riding matching Kawasaki Z1000s – a rare sight in this part of the world.  Back at the hotel, we got cheap massages and turned in for the night.

Vietnam Bike Tour: Hanoi to Ho Chi Minh City (Day 3)

DAY: 03Screenshot from 2014-06-04 23:53:31
Distance: 395km
Origin: Middle of nowhere
Endpoint: Middle of nowhere, 50km north of Phong Nha
Pho Consumed: 1

Today’s Author: Felix

 

Screenshot from 2014-06-04 23:53:40

Awakening to the blessed frigidity of the little hotel’s air conditioning, we decided we’d try to push a bit further today and set a goal for ourselves of reaching the Phong Nha national park, a UNESCO World Heritage site approximately 400km from where we spent the night.  Before heading out, we grabbed a quick bowl of beef Pho for breakfast at a shop about a hundred meters down the road from our hotel that some locals pointed us towards.  Thus far, our meal plan of “wander down the street until we see a sign for Pho” has worked remarkably well (no digestive issues, either).

 

As an aside, neither of us has the faintest idea why the Vietnamese prefer to eat hot noodle soup for breakfast in a climate where the mercury can hit 100F before noon (not to mention the near-constant 100% humidity).  Admittedly, it is incredibly delicious and quite filling, but it seems a bit… counterproductive.

After finishing our tasty-as-usual breakfast, we walked back to the hotel, loaded the bikes, and headed over to the gas station to fill up and do some basic maintenance checks.  The bikes seems to be in pretty good shape, only needing a bit of oil on each chain and for Zach’s engine.  We set off around 10:30, and quickly found ourselves on the Ho Chi Minh highway – a two-lane paved affair much more hospitable than the previous evening’s terrain.  Able to open up the bikes on the open road, we were able to easily hit 80 to 90 kph. Judging speed proved to be a bit tricky as each of our speedometers disagreed with the other, and neither agreed with the combination of Google Maps and our wristwatches.

We stopped for a quick break around 11:20 to snap some photos in an absolutely gorgeous valley, but didn’t dally for too long as the heat was already becoming miserable.

We made a longer stop around 12:30, roughly 100km in, giving us an average pace of roughly 50 kph.  While no one at the shop we stopped at spoke a lick of English, we were able to gesticulate sufficiently to procure a few bottles of wonderfully cold water and a bag of potato chips.  Unfortunately, our communication skills did not extend to having the shopkeeper take a photo of us, so we had to settle for a selfie.

2014-05-15 12.45.49

After getting back on the highway around 12:45, we made pretty good progress.  During this trip, we’ve seen all sorts of insane cargo loads on motorbikes and scooters, seemingly supported by some invisible magical force (Communist Party line: something something something indomitable will of the Vietnamese people, etc.).  However, on this particular stretch, we passed a gentleman with a full-sized refrigerator balanced on his passenger seat- secured only with his left hand!  Anyways, after about 60km, we came upon an ambulance with its sirens off going roughly the same pace as us, and we followed it for another 50km until our next stop for fuel and water, around 2:30.

DSC_0411
DSC_0418

Following our fuel and water break, we passed some more incredible scenery, including a very pretty lake and a few rivers.  One thing we noticed is that the Vietnamese seem to have a fondness for hanging out on bridges and looking at rivers.  We started seeing more trucks, including a few with massive loads of grain hanging off in every direction- picture a gigantic mushroom shape with some wheels sticking out of the bottom and you’ll get the general image.  We also passed a very tall big rig with some passengers – riding 25ft up on top of the cargo!

We did have a brief moment of rain, but managed to outrun the storm and only suffered a few minutes’ worth of drizzle – although we did stop to move our electronics into a pack, which we then put under a rain cover.

We made one more stop after another 100km or so, where we enjoyed some overly sweetened drinks of mysterious composition (no water) and attempted to top up our SIM cards as we both seemed to have exceeded our limit of 3G data service and had been consigned to the indignity of EDGE data (#firstworldproblems).  After more gesticulating with some very confused but helpful locals, we managed to put more money on our phones, but our data service still seems to be relegated to EDGE only.  We did, however, also manage to procure some delicious cookies (think reverse oreos; chocolate creme, vanilla cookies).  The town also had a pretty large and impressive-looking church that we passed on our way out.  Interestingly, there was a similar church less than 2km away in another town, but we haven’t seen any others like them before or since.

2014-05-15 12.12.37

As the light began to fail, we entered a more mountainous area with some incredible riding roads.  Kilometer after kilometer of nice 50 kph sweepers stretched out ahead of us, but we had to be careful as there were still a few trucks on the highway.  These trucks weren’t well suited to the road and presented a serious road hazard for the unwary.  As things began to get darker, we also noticed that many of these trucks had been customized with bright strips of wildly colored LEDs – certainly a different take on the trucker aesthetic compared to Smokey and the Bandit.  Also of note is the fact that we didn’t see any Westerners the entire day, including on the road – perhaps fewer people do this trip than we originally thought?

We ended the day in a tiny village along the highway, where we were directed to a perfectly serviceable guesthouse.  Thankfully, they had air conditioning – the air was still quite hot, even at 7pm.  While we didn’t quite make it to Phong Nha (turns out Google lied and it was more like 450km), but we did hit our 400km goal almost exactly.  After haggling a bit over the price of the room, we sat down to a simple dinner of soup, egg, rice, and greens.  Before we turned in, some friendly locals insisted we share a drink with them.  After a round or five of cheap but surprisingly drinkable vodka (and a lot of smiling, gesticulating, and hand-shaking), we were able to beg off and communicate that we had to be up early to hit the road again.