Eapen's FavesStuff I like from around the web | Personal repository

Alright, sorry about the delay. I was too busy celebrating the New Year. ;) I hope you're still checking in on this account.

Anyway, I think I have a bit of a unique perspective. I've seen MIT admissions from the perspective of the applicant, a student, a teacher, and now as an alumnus conducting interviews of prospective students. The fact that you mentioned MIT specifically really made me feel like I should take the time to produce a good response!

I wanted to start by writing out standard admissions advice (e.g. no one thing like SAT scores will keep you from being admitted, etc.). While all that is true, the problem you're dealing with is so much bigger than that. The problem you're coming up against is one I've seen so many of my fellow students encounter. If I could set up a wavy-fade flashback, I'd show you my freshman year.

I moved into one of the dorms at MIT thinking I was hot shit. I had, after all, just gotten into MIT. And beyond that, I had tested out of the freshman calculus and physics classes, meaning that I was able to start math "a year" ahead in differential equations and start with the advanced version of the physics 2 class we have. Registration went by easy enough and I was pleased with my decisions.

Term rolled in and I was getting crushed. I wasn't the greatest student in high school, and whenever I got poor grades I would explain them away by saying I just didn't care or I was too busy or too unmotivated or (more often than not) just cared about something else. It didn't help that I had good test performance which fed my ego and let me think I was smarter than everyone else, just relatively unmotivated. I had grossly underestimated MIT, and was left feeling so dumb.

I had the fortune of living next to a bright guy, R. R. was an advanced student, to say the least. He was a sophomore, but was already taking the most advanced graduate math classes. He came into MIT and tested out of calculus, multivariable calculus, differential equations, linear algebra, real analysis (notoriously the most difficult math class at MIT), and a slew of other math courses. And to top it all off, he was attractive, engaging, sociable, and generally had no faults that would make him mortal.

I suffered through half a semester of differential equations before my pride let me go to R. for help. And sure enough, he took my textbook for a night to review the material (he couldn't remember it all from third grade), and then he walked me through my difficulties and coached me. I ended up pulling a B+ at the end of a semester and avoiding that train wreck. The thing is, nothing he taught me involved raw brainpower. The more I learned the more I realized that the bulk of his intelligence and his performance just came from study and practice, and that the had amassed a large artillery of intellectual and mathematical tools that he had learned and trained to call upon. He showed me some of those tools, but what I really ended up learning was how to go about finding, building, and refining my own set of cognitive tools. I admired R., and I looked up to him, and while I doubt I will ever compete with his genius, I recognize that it's because of a relative lack of my conviction and an excess of his, not some accident of genetics.

It's easy to trick ourselves into thinking that "being smart" is what determines our performance. In so many ways, it's the easiest possible explanation because it demands so little of us and immediately explains away our failings. You are facing this tension without recognizing it. You are blaming your intelligence in the first two paragraphs but you undermine yourself by saying you received good grades you didn't deserve. You recognize your lack of motivation as a factor in your lack of extracurricular activities but not in your SAT scores (fun fact: the variable that correlates most strongly to SAT performance is hours of studying for the SATs). Your very last statement could just as well apply to your entire post:

But none of this has to do with my intelligence; I'm just rambling.

You got A's because you studied or because the classes were easy. You got a B probably because you were so used to understanding things that you didn't know how to deal with something that didn't come so easily. I'm guessing that early on you built the cognitive and intellectual tools to rapidly acquire and process new information, but that you've relied on those tools so much you never really developed a good set of tools for what to do when those failed. This is what happened to me, but I didn't figure it out until after I got crushed by my first semester of college. I need to ask you, has anyone ever taken the time to teach you how to study? And separately, have you learned how to study on your own in the absence of a teacher or curriculum? These are the most valuable tools you can acquire because they are the tools you will use to develop more powerful and more insightful tools. It only snowballs from there until you become like R.

MIT has an almost 97% graduation rate. That means that most of the people who get in, get through. Do you know what separates the 3% that didn't from the rest that do? I do. I've seen it so many times, and it almost happened to me. Very few people get through four years of MIT with such piss-poor performance that they don't graduate. In fact, I can't think of a single one off the top of my head. People fail to graduate from MIT because they come in, encounter problems that are harder than anything they've had to do before, and not knowing how to look for help or how to go about wrestling those problems, burn out. The students that are successful look at that challenge, wrestle with feelings of inadequacy and stupidity, and begin to take steps hiking that mountain, knowing that bruised pride is a small price to pay for getting to see the view from the top. They ask for help, they acknowledge their inadequacies. They don't blame their lack of intelligence, they blame their lack of motivation. I was lucky that I had someone to show me how to look for that motivation, and I'm hoping that I can be that person for you in some small capacity over the Internet. I was able to recover from my freshman year and go on to be very successful in my studies, even serving as a TA for my fellow students. When I was a senior, I would sit down with the freshmen in my dorm and show them the same things that had been shown to me, and I would watch them struggle with the same feelings, and overcome them. By the time I graduated MIT, I had become the person I looked up to when I first got in.

You're so young, way too young to be worried about not being smart enough. Until you're so old you start going senile, you have the opportunity to make yourself "smarter." And I put that in quotes because "smart" is really just a way of saying "has invested so much time and sweat that you make it look effortless." You feel like you are burnt out or that you are on the verge of burning out, but in reality you are on the verge of deciding whether or not you will burn out. It's scary to acknowledge that it's a decision because it puts the onus on you to to do something about it, but it's empowering because it means there is something you can do about it.

So do it.

How to be popular

Nature Physics 7, 827 (2011) doi:10.1038/nphys2147

Published online 02 November 2011

Increasingly, scientists are expected to go beyond the traditional scientific paper to explain their research to a non-specialist readership. We offer some tips on writing popular science for a general audience.

As the budgets of governments worldwide become ever tighter, it is more important than ever for scientists to be able to explain — and in some sense justify — their research to a wider audience. Yet few have received formal training in writing at all, let alone in popular writing. There are many forms such writing can take: press releases, perspective pieces for magazines, or even blogs. In Nature Physics, although we strive to ensure that everything we publish is as accessible as possible, this is of utmost importance in our News and Views section, for which we commission practising scientists to write about research that is published in Nature Physics or elsewhere in the scientific literature. On the whole, physicists do a pretty good job of explaining these advances to colleagues in other fields. Yet there are a number of aspects of popular writing that our writers find challenging. We'll describe the most common pitfalls.

For a writer, the most important person in the Universe is the reader. And the most important thing to know about the reader is that you are subject to his or her whim. Every sentence you write must maintain the reader's interest, grip them and keep them reading to the end. As veteran reporter Tim Radford points out in his widely circulated journalists' manifesto (http://go.nature.com/JGVCyi), “This is because, although you — an employee, an apostle or an apologist — may feel compelled to write, nobody has ever felt obliged to read.” And given half a chance, they will stop reading what you have written and turn the page in an instant.

The most engaging writing tells a story. A good story is a journey, and most people won't follow you on this journey unless you give them some idea of where it's headed. So before you even start, you need to decide where you want it to end. Ask yourself, what is the most surprising or significant aspect of the work you're writing about? When you first heard (or read) about the research, what about it made you think, “Wow, I didn't expect that!” It needn't be what the scientists who conducted the work think is the most important aspect: indeed, for a Nature Physics News and Views, a blog entry or similar, it is your unique perspective that readers (and editors) are interested in.

“Remember, the piece needs a beginning, a middle and an end — and, unless you're Quentin Tarantino, it needs to be presented in this order.”

The structure of the piece should reflect the journey that you've set out to describe. Remember, it needs a beginning, a middle and an end — and, unless you're Quentin Tarantino, it needs to be presented in this order. The first paragraph is the most important of the piece. It sets the stage, addressing the cardinal questions: what, why, who and how. It introduces, briefly, the wider context and motivation for the research and identifies the key challenges it is to address. It doesn't need details — that's what the middle of the piece is for.

On any journey, it's tempting to stop at other attractions on the way. For long journeys this can break the monotony, but for short journeys it's usually a distraction. Pick one key idea, one destination, and stick to it. Sometimes in writing you can set out for one place and on the way discover yourself heading towards another — that's fine, and if you discover that the new destination is more exciting than the original, don't be afraid to change your plans. But don't try to include both destinations, unless they're closely neighbouring towns, like Buda and Pest. You only have the time to reach one.

Hype doesn't help, neither do hyperbolic adjectives such as 'very', 'extremely' and 'remarkably'. It's not enough to simply assert that something is exciting, notable, novel or useful — explain why. Don't begin any sentence with 'interestingly': it is for the reader to decide whether something is interesting, it's your job to persuade them that it is. And before you describe anything as ubiquitous, pause and consider: if everyone knows it's ubiquitous — as is usually the case for ubiquitous things — there's probably no point. Cement is ubiquitous; silicon is ubiquitous; organic molecules are ubiquitous; vector calculus is ubiquitous; so what? Analogies, on the other hand, are great. But anthropomorphisms are usually terrible. Silicon doesn't have a few tricks up its sleeve: semiconductors don't perform tricks, neither do they have sleeves.

At the opening of A Brief History of Time, Stephen Hawking notes that his editor warned him that for every equation in the book its readership would be halved, and so he included only a single equation — E = mc2. Writing for physicists is less constrained, but not by much. Few physicists are afraid of equations. But most equations — apart from those presented in any undergraduate physics course — represent a short-hand that only specialists appreciate. Consequently, formulae are usually a waste of space, unless the implications of every index and every coefficient is explained in long form. Equations in popular writing are not efficient, they're lazy. It's usually much better to describe in words the key relations that they embody.

And finally, when you've crafted some beautiful prose, be ready for the edit. Unless you're writing for your own personal blog, it's unlikely that every word in your initial draft will make the final cut. In the world of professional journalism, most submissions are edited substantially; even seasoned writers regularly see their copy transformed into something quite different from their original draft. The reason is that an editor is closer to the reader than a writer. Editors set the tone and the scope, and it's the editors' vision that ultimately determines the success of a journal. So if an editor has made a change that you're not happy with, don't just change it back — it's been edited for a reason. Explain why you're not happy with the change, and try to rephrase in way that still chimes with the editor's version.

The romantic image of an über-programmer is someone who fires up Emacs, types like a machine gun, and delivers a flawless final product from scratch. A more accurate image would be someone who stares quietly into space for a few minutes and then says “Hmm. I think I’ve seen something like this before.”

Shawn and Mark wrote this article for the Storytelling for Insight workshop that will be held in Singapore tomorrow with Gary Klein and Patrick Lambe

Nick was standing in front of a wall of stories. Each A4 sheet of paper sported a single anonymous anecdote illustrating either a good or bad management behaviour, collected from Nick's company. One story had captured Nick's attention and made him agitated: "I can't believe this guy. Imagine answering a phone in an interview. My God, he even stepped out of his office to chat with someone who was just passing by."

His complaints caused others in the workshop to wander over to see what was going on. As Nick was spluttering his displeasure, Paul, one of his colleagues, jumped in: "That was my anecdote, Nick, and it was about you." Nick's faced turned red and before he could say anything, another colleague added: "It totally nails you Nick. It's spot on. You do it all the time." By now, everyone in the workshop was watching. It seemed the next few seconds would reveal Nick's true character.

Nick's face was ashen as he looked around the room. He gathered himself and then apologised to his colleagues, adding: "I can't promise you it won't happen again - I wasn't even aware I did this. But I can promise you that I'm going to make every effort to change my behaviour." And to Nick's credit, he did. At the time of the workshop, Nick was the head of sales and marketing at the company; he's now the CEO.

There was a big difference between what Nick thought he was doing and what he was actually doing. It took a story, and the willingness of his trusted colleagues to speak up, to make him aware of his poor behaviour. As a result, Nick's insight was both cognitive and emotional: cognitive in that he could rationally understand what he was doing wrong, and emotional in that he felt intense embarrassment at having discovered that the bad behaviour he had ridiculed only moments before was his own. This combination of insight and emotion created a powerful impetus in Nick to take action.

Nick is not alone in being blind to his own behaviour. We've conducted this type of story-based workshop about 50 times. First, we ask everyone to put a green dot on the most positive stories, then to put a red dot on the stand-out negative stories. The third instruction is to put a blue dot on the stories that remind the attendees of the things they do themselves. The results are always clear-cut: at a ratio of three to one, each person says they do mostly positive things, leaving the negative behaviours to those other people in the organisation. This ratio stands even when there is an overwhelming proportion of negative stories on the wall. We just don't see ourselves as doing bad things.

The same can be said of behavioural change - sometimes we start doing things differently without being fully aware of it. During one of Shawn's family's many Christmas visits to his parents, who live near the beach at Jervis Bay to the south of Sydney, Shawn's father described how he had recently had some car problems due to a bad batch of petrol. He had had to drain his little Datsun truck of all its fuel. When Shawn asked him where he had gotten the bad gas, his father said: "This service station was being refuelled by a tanker and it was probably churning up all the sediment in the underground tanks. I happened to fill up when all that muck was floating around." Then he said: "I will never again fill up at a service station if I see a tanker parked there."

Shawn was pretty sure his dad wasn't explicitly aware of the insight he had just had. It's an example of how insights can be cognitively invisible but still result in behavioural change, and helps explain why it can be so difficult to determine why we do what we do.

Practice 1: Collect a set of anecdotes on a topic of interest. Bring together your organisation's decision makers to seek out the repeating patterns in the stories - the behaviours. You will be surprised by the insights that emerge.

*****

In 1419 the city of Florence had a beautiful cathedral, but it lacked a dome. So the city's wool merchants guild, which had been responsible for creating the cathedral, held a competition to find someone who could design and build a mammoth cupola for the Basilica di Santa Maria del Fiore. One of the entrants was Filipo Brunelleschi, a goldsmith who had come up with an ingenious design but was too afraid to fully present his idea in case it was stolen. The judges' frustrations with Brunelleschi grew and grew as he repeatedly refused to provide the detail needed to assess his entry, or an explanation for his reluctance. Then, one day, Brunelleschi suggested that each competitor take part in a test of creative skill, with the winner to take out the design competition.

The test involved taking a boiled egg and trying to balance it upright on a marble bench. Each competitor duly tried and failed. Then Brunelleschi stepped up and slammed his egg into the marble, flattening one end of it and leaving it standing perfectly upright on the bench. The selection committee discounted his performance, saying that anyone could work out how to do what he had just done. "That's exactly my point," Brunelleschi retorted. "When you see my design, you will mistake its elegant simplicity for something that is obvious - but only in hindsight." With that, he obtained a guarantee to keep his design a secret and eventually won the competition.

Now, it's most likely that the egg story is a 15th-century urban myth with as much veracity as the story of the apple that fell on Isaac Newton's head. The lesson, however, is clear: once an insight has been gained, it can seem rather obvious. One person's insight is another's common knowledge. But there is something else to learn from this story. Rather than tell the judges straight out that he had concerns about sharing his design, and so risk their denial, Brunelleschi created a situation, the egg test, that allowed the judges to work this out for themselves. In effect, Brunelleschi triggered a new story that solved his problem.

We saw the power of people working things out for themselves in Nick's story above. People rarely like to be told what or how to think. There is even a psychological predisposition that partly explains this response. The confirmation bias has it that if a person has a strong view on a topic and you attempt to argue for an alternative viewpoint, the person merely strengthens their view; they dig their heels in. To become open to changing their mind, they need to experience your new way of thinking either first-hand or vicariously through a story.

Robert Kegan, a Harvard professor and psychologist, explains in his book Immunity to Change that we need new stories to see the world differently, and that these stories come from new experiences.

This idea was made clear to Shawn while he was conducting a workshop with 80 professors at Melbourne University on ways to improve collaboration. He began to make the point that two important behaviours for good collaboration were to make and keep promises, and to speak your mind to anyone on the team with respect and good intent. But as he spoke, he noticed a woman at the back of the room who was sitting with her arms crossed, shaking her head, clearly very unhappy with what Shawn was saying. So he stopped his presentation and asked the woman if she would like to share what she was thinking with the rest of the group. Practically before Shawn had finished his request, she said, "There is no way in the world you can be open and honest with a senior professor around here." Before Shawn could comment, she went on to tell a mini story: "I once did what you are suggesting and I had to move departments."

Now, no amount of clever argument or telling of familiar stories would have changed that person's mind. She had obviously had an incredibly bad experience. The only way to help her gain a new insight would be to create an experience with a different result to what she was expecting. She would then have a new story that would in turn guide her future behaviour.

Practice 2: Create new experiences for your team where they come to understand and feel a totally new way of thinking. Importantly, give them an opportunity to gain the insight for themselves. Simply telling them about it is often insufficient. It is much better for them to act their way into a new form of thinking.

*****

A couple of years ago, a university library was preparing to move to a new, purpose-built, ultra-modern building. The move required a huge number of things to change, including the library's culture, and we were invited to help it with this aspect. The first thing we did was to collect stories from the library's employees which illustrated the current culture and values. Then we gathered everyone together for a workshop to identify the patterns in those stories.

At one point in the sense-making workshop, 10 librarians were looking at a set of anecdotes about their value of 'excellence'. After reviewing their cluster of post-it notes, they concluded that the key issue was that they needed more training. They refused to change this view despite Mark's gentle prompting that there might be something going on at a deeper level. Then Mark suggested they use a story spine to tell the story of 'training' in the library.

A story spine is just a simple story structure. Here is the one we suggested the librarians use:

In the past ...
Every day ...
But one day ...
Because of that ... (repeat three times or as often as necessary)
Until finally ...
Ever since then ...
And the moral of the story is ... (optional)

The librarians then set about creating a story that explained what was happening in the organisation around training. The story they produced was about a woman named Sue (not an actual person but a character representative of a type of person in the library) who had a bad habit of talking behind people's backs. Sue was always bitching about people on the one hand, but always said the right things to the right people on the other hand. Then, after Sue was promoted, people realised she couldn't do the job and they started bitching about her. One day, one of the staff, who had left because Sue was mean to him, ran over her in his BMW at some traffic lights. Many people danced and were happy.

The librarians were shocked at this story. They looked at each other and, almost in unison, said: "We don't have a training problem in our library. We have a bitching problem." And right there and then they committed themselves to tackling bitching, which they ultimately did.

Sometimes an insight is sitting just under the surface of people's awareness, waiting to be named - as soon as it is, everyone knows it to be true. The story structure creates a safe way to talk about these types of sensitive issues because the authors of the story are illustrating the behaviours without personalising the actions.

You just haven't made sense of something until you can tell a story about it.

Practice 3: Whenever your team gets stuck and thinks there is something just under the surface of everyday practice that's creating the problems, pull out the story spine and get them to create some stories to explain what's happening.

*****

One day an expert photocopier technician goes to help a new team member with a problem he's been trying to solve for a while but without any luck. The devices in question are not ordinary photocopiers but monster machines designed for high throughput. The new technician starts by telling the expert the story of what has happened so far, then they try a few things. When the machines display an E053 error, the expert groans and says, "I remember the first one of these I ever had ... if it is what I think it is. I got an E053 and immediately started replacing the dead shorted dicrotrons. They were blowing the circuit breaker. But as soon as I did this I created a 24-volt interlock problem and you can chase that one for ever and NEVER find out what it is. I happened to pull up the dC20 log and I could see I was getting hits on the XER board. It was an XER failure. So I replaced it, then the dicrotrons, and stress-tested the bugger and the real error code displayed. You can't believe what the machine tells you."

This story is an adaptation of one of the marvellous anecdotes Julian Orr recorded while conducting his ethnographic study of Xerox photocopier technicians for his book Talking about Machines. The stories that the two technicians tell each other establish the common ground that allows them to understand what is happening when they start diagnosing the problem. It's impossible to notice anomalies unless you have a story that represents what is normal. Then, when something happens that doesn't meet set expectations, a new story is needed to make sense of the new facts. In this example, a single story is told that holds the answer. More typically, many stories are told, each one compared and contrasted with the situation at hand. Each story subtly rearranges the facts until an insight emerges.

We can systematically create insights with stories by taking scenarios from other industries or companies and applying them to the problem at hand. The IT research firm Gartner has noticed that some of the best-performing organisations actively seek out business models - fundamentally, stories of how businesses create value - from industries that are different to them yet have similar business characteristics. For example, an airport examined Disneyland's amusement parks for insights because the objective of both businesses is to get lots of people who are milling in a confined area to eventually line up: in one case to board a flight, and in the other to board a ride. Insights emerged for the airport when it viewed itself as an amusement park.

Practice 4: When diagnosing a problem (strategy development is a good example), encourage everyone to share stories that shed light on what's happening. Then systematically seek out stories from other industries that are structurally similar to yours. Get used to finding and telling these stories.

*****

The stories we tell ourselves form the foundation for how we perceive and receive new data. Inputs either accord with our story, pull at its edges or turn it upside down. A somersault is usually an indicator of real insight. The question then becomes: Do we bury it or do we actively try and create a new story to explain it?

Stories are also fundamental in creating common ground among groups, and as we saw with the Xerox technicians, they act as a thinking device to help us reorder and arrange the facts we are receiving. Stories are essential for diagnosing and making sense of what we notice.

Stories also hold up a mirror to our own behaviours. Not only do they help us to see something new, they also generate the emotion that motivates us to take action as a result of such insights.

Lastly, we can take stories further and apply them systematically by seeking out analogies from other industries. These foreign stories are the equivalent of putting on a new pair of glasses that help us interpret the familiar in a totally new light. As a result, insight occurs.

This week, Hewlett-Packard (where I am on the board) announced that it is exploring jettisoning its struggling PC business in favor of investing more heavily in software, where it sees better potential for growth. Meanwhile, Google plans to buy up the cellphone handset maker Motorola Mobility. Both moves surprised the tech world. But both moves are also in line with a trend I've observed, one that makes me optimistic about the future growth of the American and world economies, despite the recent turmoil in the stock market.

In an interview with WSJ's Kevin Delaney, Groupon and LinkedIn investor Marc Andreessen insists that the recent popularity of tech companies does not constitute a bubble. He also stressed that both Apple and Google are undervalued and that "the market doesn't like tech."

In short, software is eating the world.

More than 10 years after the peak of the 1990s dot-com bubble, a dozen or so new Internet companies like Facebook and Twitter are sparking controversy in Silicon Valley, due to their rapidly growing private market valuations, and even the occasional successful IPO. With scars from the heyday of Webvan and Pets.com still fresh in the investor psyche, people are asking, "Isn't this just a dangerous new bubble?"

I, along with others, have been arguing the other side of the case. (I am co-founder and general partner of venture capital firm Andreessen-Horowitz, which has invested in Facebook, Groupon, Skype, Twitter, Zynga, and Foursquare, among others. I am also personally an investor in LinkedIn.) We believe that many of the prominent new Internet companies are building real, high-growth, high-margin, highly defensible businesses.

QuickHoney

Today's stock market actually hates technology, as shown by all-time low price/earnings ratios for major public technology companies. Apple, for example, has a P/E ratio of around 15.2—about the same as the broader stock market, despite Apple's immense profitability and dominant market position (Apple in the last couple weeks became the biggest company in America, judged by market capitalization, surpassing Exxon Mobil). And, perhaps most telling, you can't have a bubble when people are constantly screaming "Bubble!"

But too much of the debate is still around financial valuation, as opposed to the underlying intrinsic value of the best of Silicon Valley's new companies. My own theory is that we are in the middle of a dramatic and broad technological and economic shift in which software companies are poised to take over large swathes of the economy.

More and more major businesses and industries are being run on software and delivered as online services—from movies to agriculture to national defense. Many of the winners are Silicon Valley-style entrepreneurial technology companies that are invading and overturning established industry structures. Over the next 10 years, I expect many more industries to be disrupted by software, with new world-beating Silicon Valley companies doing the disruption in more cases than not.

QuickHoney

Why is this happening now?

Six decades into the computer revolution, four decades since the invention of the microprocessor, and two decades into the rise of the modern Internet, all of the technology required to transform industries through software finally works and can be widely delivered at global scale.

Over two billion people now use the broadband Internet, up from perhaps 50 million a decade ago, when I was at Netscape, the company I co-founded. In the next 10 years, I expect at least five billion people worldwide to own smartphones, giving every individual with such a phone instant access to the full power of the Internet, every moment of every day.

On the back end, software programming tools and Internet-based services make it easy to launch new global software-powered start-ups in many industries—without the need to invest in new infrastructure and train new employees. In 2000, when my partner Ben Horowitz was CEO of the first cloud computing company, Loudcloud, the cost of a customer running a basic Internet application was approximately $150,000 a month. Running that same application today in Amazon's cloud costs about $1,500 a month.

QuickHoney

With lower start-up costs and a vastly expanded market for online services, the result is a global economy that for the first time will be fully digitally wired—the dream of every cyber-visionary of the early 1990s, finally delivered, a full generation later.

Perhaps the single most dramatic example of this phenomenon of software eating a traditional business is the suicide of Borders and corresponding rise of Amazon. In 2001, Borders agreed to hand over its online business to Amazon under the theory that online book sales were non-strategic and unimportant.

Oops.

Today, the world's largest bookseller, Amazon, is a software company—its core capability is its amazing software engine for selling virtually everything online, no retail stores necessary. On top of that, while Borders was thrashing in the throes of impending bankruptcy, Amazon rearranged its web site to promote its Kindle digital books over physical books for the first time. Now even the books themselves are software.

Today's largest video service by number of subscribers is a software company: Netflix. How Netflix eviscerated Blockbuster is an old story, but now other traditional entertainment providers are facing the same threat. Comcast, Time Warner and others are responding by transforming themselves into software companies with efforts such as TV Everywhere, which liberates content from the physical cable and connects it to smartphones and tablets.

Today's dominant music companies are software companies, too: Apple's iTunes, Spotify and Pandora. Traditional record labels increasingly exist only to provide those software companies with content. Industry revenue from digital channels totaled $4.6 billion in 2010, growing to 29% of total revenue from 2% in 2004.

Today's fastest growing entertainment companies are videogame makers—again, software—with the industry growing to $60 billion from $30 billion five years ago. And the fastest growing major videogame company is Zynga (maker of games including FarmVille), which delivers its games entirely online. Zynga's first-quarter revenues grew to $235 million this year, more than double revenues from a year earlier. Rovio, maker of Angry Birds, is expected to clear $100 million in revenue this year (the company was nearly bankrupt when it debuted the popular game on the iPhone in late 2009). Meanwhile, traditional videogame powerhouses like Electronic Arts and Nintendo have seen revenues stagnate and fall.

The best new movie production company in many decades, Pixar, was a software company. Disney—Disney!—had to buy Pixar, a software company, to remain relevant in animated movies.

Photography, of course, was eaten by software long ago. It's virtually impossible to buy a mobile phone that doesn't include a software-powered camera, and photos are uploaded automatically to the Internet for permanent archiving and global sharing. Companies like Shutterfly, Snapfish and Flickr have stepped into Kodak's place.

Today's largest direct marketing platform is a software company—Google. Now it's been joined by Groupon, Living Social, Foursquare and others, which are using software to eat the retail marketing industry. Groupon generated over $700 million in revenue in 2010, after being in business for only two years.

Today's fastest growing telecom company is Skype, a software company that was just bought by Microsoft for $8.5 billion. CenturyLink, the third largest telecom company in the U.S., with a $20 billion market cap, had 15 million access lines at the end of June 30—declining at an annual rate of about 7%. Excluding the revenue from its Qwest acquisition, CenturyLink's revenue from these legacy services declined by more than 11%. Meanwhile, the two biggest telecom companies, AT&T and Verizon, have survived by transforming themselves into software companies, partnering with Apple and other smartphone makers.

QuickHoney

LinkedIn is today's fastest growing recruiting company. For the first time ever, on LinkedIn, employees can maintain their own resumes for recruiters to search in real time—giving LinkedIn the opportunity to eat the lucrative $400 billion recruiting industry.

Software is also eating much of the value chain of industries that are widely viewed as primarily existing in the physical world. In today's cars, software runs the engines, controls safety features, entertains passengers, guides drivers to destinations and connects each car to mobile, satellite and GPS networks. The days when a car aficionado could repair his or her own car are long past, due primarily to the high software content. The trend toward hybrid and electric vehicles will only accelerate the software shift—electric cars are completely computer controlled. And the creation of software-powered driverless cars is already under way at Google and the major car companies.

Today's leading real-world retailer, Wal-Mart, uses software to power its logistics and distribution capabilities, which it has used to crush its competition. Likewise for FedEx, which is best thought of as a software network that happens to have trucks, planes and distribution hubs attached. And the success or failure of airlines today and in the future hinges on their ability to price tickets and optimize routes and yields correctly—with software.

Oil and gas companies were early innovators in supercomputing and data visualization and analysis, which are crucial to today's oil and gas exploration efforts. Agriculture is increasingly powered by software as well, including satellite analysis of soils linked to per-acre seed selection software algorithms.

The financial services industry has been visibly transformed by software over the last 30 years. Practically every financial transaction, from someone buying a cup of coffee to someone trading a trillion dollars of credit default derivatives, is done in software. And many of the leading innovators in financial services are software companies, such as Square, which allows anyone to accept credit card payments with a mobile phone, and PayPal, which generated more than $1 billion in revenue in the second quarter of this year, up 31% over the previous year.

Health care and education, in my view, are next up for fundamental software-based transformation. My venture capital firm is backing aggressive start-ups in both of these gigantic and critical industries. We believe both of these industries, which historically have been highly resistant to entrepreneurial change, are primed for tipping by great new software-centric entrepreneurs.

Even national defense is increasingly software-based. The modern combat soldier is embedded in a web of software that provides intelligence, communications, logistics and weapons guidance. Software-powered drones launch airstrikes without putting human pilots at risk. Intelligence agencies do large-scale data mining with software to uncover and track potential terrorist plots.

Companies in every industry need to assume that a software revolution is coming. This includes even industries that are software-based today. Great incumbent software companies like Oracle and Microsoft are increasingly threatened with irrelevance by new software offerings like Salesforce.com and Android (especially in a world where Google owns a major handset maker).

In some industries, particularly those with a heavy real-world component such as oil and gas, the software revolution is primarily an opportunity for incumbents. But in many industries, new software ideas will result in the rise of new Silicon Valley-style start-ups that invade existing industries with impunity. Over the next 10 years, the battles between incumbents and software-powered insurgents will be epic. Joseph Schumpeter, the economist who coined the term "creative destruction," would be proud.

QuickHoney

And while people watching the values of their 401(k)s bounce up and down the last few weeks might doubt it, this is a profoundly positive story for the American economy, in particular. It's not an accident that many of the biggest recent technology companies—including Google, Amazon, eBay and more—are American companies. Our combination of great research universities, a pro-risk business culture, deep pools of innovation-seeking equity capital and reliable business and contract law is unprecedented and unparalleled in the world.

Still, we face several challenges.

First of all, every new company today is being built in the face of massive economic headwinds, making the challenge far greater than it was in the relatively benign '90s. The good news about building a company during times like this is that the companies that do succeed are going to be extremely strong and resilient. And when the economy finally stabilizes, look out—the best of the new companies will grow even faster.

Secondly, many people in the U.S. and around the world lack the education and skills required to participate in the great new companies coming out of the software revolution. This is a tragedy since every company I work with is absolutely starved for talent. Qualified software engineers, managers, marketers and salespeople in Silicon Valley can rack up dozens of high-paying, high-upside job offers any time they want, while national unemployment and underemployment is sky high. This problem is even worse than it looks because many workers in existing industries will be stranded on the wrong side of software-based disruption and may never be able to work in their fields again. There's no way through this problem other than education, and we have a long way to go.

Finally, the new companies need to prove their worth. They need to build strong cultures, delight their customers, establish their own competitive advantages and, yes, justify their rising valuations. No one should expect building a new high-growth, software-powered company in an established industry to be easy. It's brutally difficult.

I'm privileged to work with some of the best of the new breed of software companies, and I can tell you they're really good at what they do. If they perform to my and others' expectations, they are going to be highly valuable cornerstone companies in the global economy, eating markets far larger than the technology industry has historically been able to pursue.

Instead of constantly questioning their valuations, let's seek to understand how the new generation of technology companies are doing what they do, what the broader consequences are for businesses and the economy and what we can collectively do to expand the number of innovative new software companies created in the U.S. and around the world.

That's the big opportunity. I know where I'm putting my money.

Three men doing time in Israeli prisons recently appeared before a parole board consisting of a judge, a criminologist and a social worker. The three prisoners had completed at least two-thirds of their sentences, but the parole board granted freedom to only one of them. Guess which one:

¶ Case 1 (heard at 8:50 a.m.): An Arab Israeli serving a 30-month sentence for fraud.

¶ Case 2 (heard at 3:10 p.m.): A Jewish Israeli serving a 16-month sentence for assault.

¶ Case 3 (heard at 4:25 p.m.): An Arab Israeli serving a 30-month sentence for fraud.

¶ There was a pattern to the parole board’s decisions, but it wasn’t related to the men’s ethnic backgrounds, crimes or sentences. It was all about timing, as researchers discovered by analyzing more than 1,100 decisions over the course of a year. Judges, who would hear the prisoners’ appeals and then get advice from the other members of the board, approved parole in about a third of the cases, but the probability of being paroled fluctuated wildly throughout the day. Prisoners who appeared early in the morning received parole about 70 percent of the time, while those who appeared late in the day were paroled less than 10 percent of the time.

¶ The odds favored the prisoner who appeared at 8:50 a.m. — and he did in fact receive parole. But even though the other Arab Israeli prisoner was serving the same sentence for the same crime — fraud — the odds were against him when he appeared (on a different day) at 4:25 in the afternoon. He was denied parole, as was the Jewish Israeli prisoner at 3:10 p.m, whose sentence was shorter than that of the man who was released. They were just asking for parole at the wrong time of day.

¶ There was nothing malicious or even unusual about the judges’ behavior, which was reported earlier this year by Jonathan Levav of Stanford and Shai Danziger of Ben-Gurion University. The judges’ erratic judgment was due to the occupational hazard of being, as George W. Bush once put it, “the decider.” The mental work of ruling on case after case, whatever the individual merits, wore them down. This sort of decision fatigue can make quarterbacks prone to dubious choices late in the game and C.F.O.’s prone to disastrous dalliances late in the evening. It routinely warps the judgment of everyone, executive and nonexecutive, rich and poor — in fact, it can take a special toll on the poor. Yet few people are even aware of it, and researchers are only beginning to understand why it happens and how to counteract it.

¶ Decision fatigue helps explain why ordinarily sensible people get angry at colleagues and families, splurge on clothes, buy junk food at the supermarket and can’t resist the dealer’s offer to rustproof their new car. No matter how rational and high-minded you try to be, you can’t make decision after decision without paying a biological price. It’s different from ordinary physical fatigue — you’re not consciously aware of being tired — but you’re low on mental energy. The more choices you make throughout the day, the harder each one becomes for your brain, and eventually it looks for shortcuts, usually in either of two very different ways. One shortcut is to become reckless: to act impulsively instead of expending the energy to first think through the consequences. (Sure, tweet that photo! What could go wrong?) The other shortcut is the ultimate energy saver: do nothing. Instead of agonizing over decisions, avoid any choice. Ducking a decision often creates bigger problems in the long run, but for the moment, it eases the mental strain. You start to resist any change, any potentially risky move — like releasing a prisoner who might commit a crime. So the fatigued judge on a parole board takes the easy way out, and the prisoner keeps doing time.

¶ Decision fatigue is the newest discovery involving a phenomenon called ego depletion, a term coined by the social psychologist Roy F. Baumeister in homage to a Freudian hypothesis. Freud speculated that the self, or ego, depended on mental activities involving the transfer of energy. He was vague about the details, though, and quite wrong about some of them (like his idea that artists “sublimate” sexual energy into their work, which would imply that adultery should be especially rare at artists’ colonies). Freud’s energy model of the self was generally ignored until the end of the century, when Baumeister began studying mental discipline in a series of experiments, first at Case Western and then at Florida State University.

¶ These experiments demonstrated that there is a finite store of mental energy for exerting self-control. When people fended off the temptation to scarf down M&M’s or freshly baked chocolate-chip cookies, they were then less able to resist other temptations. When they forced themselves to remain stoic during a tearjerker movie, afterward they gave up more quickly on lab tasks requiring self-discipline, like working on a geometry puzzle or squeezing a hand-grip exerciser. Willpower turned out to be more than a folk concept or a metaphor. It really was a form of mental energy that could be exhausted. The experiments confirmed the 19th-century notion of willpower being like a muscle that was fatigued with use, a force that could be conserved by avoiding temptation. To study the process of ego depletion, researchers concentrated initially on acts involving self-control ­— the kind of self-discipline popularly associated with willpower, like resisting a bowl of ice cream. They weren’t concerned with routine decision-making, like choosing between chocolate and vanilla, a mental process that they assumed was quite distinct and much less strenuous. Intuitively, the chocolate-vanilla choice didn’t appear to require willpower.

¶ But then a postdoctoral fellow, Jean Twenge, started working at Baumeister’s laboratory right after planning her wedding. As Twenge studied the results of the lab’s ego-depletion experiments, she remembered how exhausted she felt the evening she and her fiancé went through the ritual of registering for gifts. Did they want plain white china or something with a pattern? Which brand of knives? How many towels? What kind of sheets? Precisely how many threads per square inch?

¶ “By the end, you could have talked me into anything,” Twenge told her new colleagues. The symptoms sounded familiar to them too, and gave them an idea. A nearby department store was holding a going-out-of-business sale, so researchers from the lab went off to fill their car trunks with simple products — not exactly wedding-quality gifts, but sufficiently appealing to interest college students. When they came to the lab, the students were told they would get to keep one item at the end of the experiment, but first they had to make a series of choices. Would they prefer a pen or a candle? A vanilla-scented candle or an almond-scented one? A candle or a T-shirt? A black T-shirt or a red T-shirt? A control group, meanwhile — let’s call them the nondeciders — spent an equally long period contemplating all these same products without having to make any choices. They were asked just to give their opinion of each product and report how often they had used such a product in the last six months.

¶ Afterward, all the participants were given one of the classic tests of self-control: holding your hand in ice water for as long as you can. The impulse is to pull your hand out, so self-discipline is needed to keep the hand underwater. The deciders gave up much faster; they lasted 28 seconds, less than half the 67-second average of the nondeciders. Making all those choices had apparently sapped their willpower, and it wasn’t an isolated effect. It was confirmed in other experiments testing students after they went through exercises like choosing courses from the college catalog.

¶ For a real-world test of their theory, the lab’s researchers went into that great modern arena of decision making: the suburban mall. They interviewed shoppers about their experiences in the stores that day and then asked them to solve some simple arithmetic problems. The researchers politely asked them to do as many as possible but said they could quit at any time. Sure enough, the shoppers who had already made the most decisions in the stores gave up the quickest on the math problems. When you shop till you drop, your willpower drops, too.

¶ Any decision, whether it’s what pants to buy or whether to start a war, can be broken down into what psychologists call the Rubicon model of action phases, in honor of the river that separated Italy from the Roman province of Gaul. When Caesar reached it in 49 B.C., on his way home after conquering the Gauls, he knew that a general returning to Rome was forbidden to take his legions across the river with him, lest it be considered an invasion of Rome. Waiting on the Gaul side of the river, he was in the “predecisional phase” as he contemplated the risks and benefits of starting a civil war. Then he stopped calculating and crossed the Rubicon, reaching the “postdecisional phase,” which Caesar defined much more felicitously: “The die is cast.”

¶ The whole process could deplete anyone’s willpower, but which phase of the decision-making process was most fatiguing? To find out, Kathleen Vohs, a former colleague of Baumeister’s now at the University of Minnesota, performed an experiment using the self-service Web site of Dell Computers. One group in the experiment carefully studied the advantages and disadvantages of various features available for a computer — the type of screen, the size of the hard drive, etc. — without actually making a final decision on which ones to choose. A second group was given a list of predetermined specifications and told to configure a computer by going through the laborious, step-by-step process of locating the specified features among the arrays of options and then clicking on the right ones. The purpose of this was to duplicate everything that happens in the postdecisional phase, when the choice is implemented. The third group had to figure out for themselves which features they wanted on their computers and go through the process of choosing them; they didn’t simply ponder options (like the first group) or implement others’ choices (like the second group). They had to cast the die, and that turned out to be the most fatiguing task of all. When self-control was measured, they were the one who were most depleted, by far.

¶ The experiment showed that crossing the Rubicon is more tiring than anything that happens on either bank — more mentally fatiguing than sitting on the Gaul side contemplating your options or marching on Rome once you’ve crossed. As a result, someone without Caesar’s willpower is liable to stay put. To a fatigued judge, denying parole seems like the easier call not only because it preserves the status quo and eliminates the risk of a parolee going on a crime spree but also because it leaves more options open: the judge retains the option of paroling the prisoner at a future date without sacrificing the option of keeping him securely in prison right now. Part of the resistance against making decisions comes from our fear of giving up options. The word “decide” shares an etymological root with “homicide,” the Latin word “caedere,” meaning “to cut down” or “to kill,” and that loss looms especially large when decision fatigue sets in.

¶ Once you’re mentally depleted, you become reluctant to make trade-offs, which involve a particularly advanced and taxing form of decision making. In the rest of the animal kingdom, there aren’t a lot of protracted negotiations between predators and prey. To compromise is a complex human ability and therefore one of the first to decline when willpower is depleted. You become what researchers call a cognitive miser, hoarding your energy. If you’re shopping, you’re liable to look at only one dimension, like price: just give me the cheapest. Or you indulge yourself by looking at quality: I want the very best (an especially easy strategy if someone else is paying). Decision fatigue leaves you vulnerable to marketers who know how to time their sales, as Jonathan Levav, the Stanford professor, demonstrated in experiments involving tailored suits and new cars.

¶ The idea for these experiments also happened to come in the preparations for a wedding, a ritual that seems to be the decision-fatigue equivalent of Hell Week. At his fiancée’s suggestion, Levav visited a tailor to have a bespoke suit made and began going through the choices of fabric, type of lining and style of buttons, lapels, cuffs and so forth.

¶ “By the time I got through the third pile of fabric swatches, I wanted to kill myself,” Levav recalls. “I couldn’t tell the choices apart anymore. After a while my only response to the tailor became ‘What do you recommend?’ I just couldn’t take it.”

¶ Levav ended up not buying any kind of bespoke suit (the $2,000 price made that decision easy enough), but he put the experience to use in a pair of experiments conducted with Mark Heitmann, then at Christian-Albrechts University in Germany; Andreas Herrmann, at the University of St. Gallen in Switzerland; and Sheena Iyengar, of Columbia. One involved asking M.B.A. students in Switzerland to choose a bespoke suit; the other was conducted at German car dealerships, where customers ordered options for their new sedans. The car buyers — and these were real customers spending their own money — had to choose, for instance, among 4 styles of gearshift knobs, 13 kinds of wheel rims, 25 configurations of the engine and gearbox and a palette of 56 colors for the interior.

¶ As they started picking features, customers would carefully weigh the choices, but as decision fatigue set in, they would start settling for whatever the default option was. And the more tough choices they encountered early in the process — like going through those 56 colors to choose the precise shade of gray or brown — the quicker people became fatigued and settled for the path of least resistance by taking the default option. By manipulating the order of the car buyers’ choices, the researchers found that the customers would end up settling for different kinds of options, and the average difference totaled more than 1,500 euros per car (about $2,000 at the time). Whether the customers paid a little extra for fancy wheel rims or a lot extra for a more powerful engine depended on when the choice was offered and how much willpower was left in the customer.

¶ Similar results were found in the experiment with custom-made suits: once decision fatigue set in, people tended to settle for the recommended option. When they were confronted early on with the toughest decisions — the ones with the most options, like the 100 fabrics for the suit — they became fatigued more quickly and also reported enjoying the shopping experience less.

¶ Shopping can be especially tiring for the poor, who have to struggle continually with trade-offs. Most of us in America won’t spend a lot of time agonizing over whether we can afford to buy soap, but it can be a depleting choice in rural India. Dean Spears, an economist at Princeton, offered people in 20 villages in Rajasthan in northwestern India the chance to buy a couple of bars of brand-name soap for the equivalent of less than 20 cents. It was a steep discount off the regular price, yet even that sum was a strain for the people in the 10 poorest villages. Whether or not they bought the soap, the act of making the decision left them with less willpower, as measured afterward in a test of how long they could squeeze a hand grip. In the slightly more affluent villages, people’s willpower wasn’t affected significantly. Because they had more money, they didn’t have to spend as much effort weighing the merits of the soap versus, say, food or medicine.

¶ Spears and other researchers argue that this sort of decision fatigue is a major — and hitherto ignored — factor in trapping people in poverty. Because their financial situation forces them to make so many trade-offs, they have less willpower to devote to school, work and other activities that might get them into the middle class. It’s hard to know exactly how important this factor is, but there’s no doubt that willpower is a special problem for poor people. Study after study has shown that low self-control correlates with low income as well as with a host of other problems, including poor achievement in school, divorce, crime, alcoholism and poor health. Lapses in self-control have led to the notion of the “undeserving poor” — epitomized by the image of the welfare mom using food stamps to buy junk food — but Spears urges sympathy for someone who makes decisions all day on a tight budget. In one study, he found that when the poor and the rich go shopping, the poor are much more likely to eat during the shopping trip. This might seem like confirmation of their weak character — after all, they could presumably save money and improve their nutrition by eating meals at home instead of buying ready-to-eat snacks like Cinnabons, which contribute to the higher rate of obesity among the poor. But if a trip to the supermarket induces more decision fatigue in the poor than in the rich — because each purchase requires more mental trade-offs — by the time they reach the cash register, they’ll have less willpower left to resist the Mars bars and Skittles. Not for nothing are these items called impulse purchases.

¶ And this isn’t the only reason that sweet snacks are featured prominently at the cash register, just when shoppers are depleted after all their decisions in the aisles. With their willpower reduced, they’re more likely to yield to any kind of temptation, but they’re especially vulnerable to candy and soda and anything else offering a quick hit of sugar. While supermarkets figured this out a long time ago, only recently did researchers discover why.

¶ The discovery was an accident resulting from a failed experiment at Baumeister’s lab. The researchers set out to test something called the Mardi Gras theory — the notion that you could build up willpower by first indulging yourself in pleasure, the way Mardi Gras feasters do just before the rigors of Lent. In place of a Fat Tuesday breakfast, the chefs in the lab at Florida State whipped up lusciously thick milkshakes for a group of subjects who were resting in between two laboratory tasks requiring willpower. Sure enough, the delicious shakes seemed to strengthen willpower by helping people perform better than expected on the next task. So far, so good. But the experiment also included a control group of people who were fed a tasteless concoction of low-fat dairy glop. It provided them with no pleasure, yet it produced similar improvements in self-control. The Mardi Gras theory looked wrong. Besides tragically removing an excuse for romping down the streets of New Orleans, the result was embarrassing for the researchers. Matthew Gailliot, the graduate student who ran the study, stood looking down at his shoes as he told Baumeister about the fiasco.

¶ Baumeister tried to be optimistic. Maybe the study wasn’t a failure. Something had happened, after all. Even the tasteless glop had done the job, but how? If it wasn’t the pleasure, could it be the calories? At first the idea seemed a bit daft. For decades, psychologists had been studying performance on mental tasks without worrying much about the results being affected by dairy-product consumption. They liked to envision the human mind as a computer, focusing on the way it processed information. In their eagerness to chart the human equivalent of the computer’s chips and circuits, most psychologists neglected one mundane but essential part of the machine: the power supply. The brain, like the rest of the body, derived energy from glucose, the simple sugar manufactured from all kinds of foods. To establish cause and effect, researchers at Baumeister’s lab tried refueling the brain in a series of experiments involving lemonade mixed either with sugar or with a diet sweetener. The sugary lemonade provided a burst of glucose, the effects of which could be observed right away in the lab; the sugarless variety tasted quite similar without providing the same burst of glucose. Again and again, the sugar restored willpower, but the artificial sweetener had no effect. The glucose would at least mitigate the ego depletion and sometimes completely reverse it. The restored willpower improved people’s self-control as well as the quality of their decisions: they resisted irrational bias when making choices, and when asked to make financial decisions, they were more likely to choose the better long-term strategy instead of going for a quick payoff. The ego-depletion effect was even demonstrated with dogs in two studies by Holly Miller and Nathan DeWall at the University of Kentucky. After obeying sit and stay commands for 10 minutes, the dogs performed worse on self-control tests and were also more likely to make the dangerous decision to challenge another dog’s turf. But a dose of glucose restored their willpower.

¶ Despite this series of findings, brain researchers still had some reservations about the glucose connection. Skeptics pointed out that the brain’s overall use of energy remains about the same regardless of what a person is doing, which doesn’t square easily with the notion of depleted energy affecting willpower. Among the skeptics was Todd Heatherton, who worked with Baumeister early in his career and eventually wound up at Dartmouth, where he became a pioneer of what is called social neuroscience: the study of links between brain processes and social behavior. He believed in ego depletion, but he didn’t see how this neural process could be caused simply by variations in glucose levels. To observe the process — and to see if it could be reversed by glucose — he and his colleagues recruited 45 female dieters and recorded images of their brains as they reacted to pictures of food. Next the dieters watched a comedy video while forcing themselves to suppress their laughter — a standard if cruel way to drain mental energy and induce ego depletion. Then they were again shown pictures of food, and the new round of brain scans revealed the effects of ego depletion: more activity in the nucleus accumbens, the brain’s reward center, and a corresponding decrease in the amygdala, which ordinarily helps control impulses. The food’s appeal registered more strongly while impulse control weakened — not a good combination for anyone on a diet. But suppose people in this ego-depleted state got a quick dose of glucose? What would a scan of their brains reveal?

¶ The results of the experiment were announced in January, during Heatherton’s speech accepting the leadership of the Society for Personality and Social Psychology, the world’s largest group of social psychologists. In his presidential address at the annual meeting in San Antonio, Heatherton reported that administering glucose completely reversed the brain changes wrought by depletion — a finding, he said, that thoroughly surprised him. Heatherton’s results did much more than provide additional confirmation that glucose is a vital part of willpower; they helped solve the puzzle over how glucose could work without global changes in the brain’s total energy use. Apparently ego depletion causes activity to rise in some parts of the brain and to decline in others. Your brain does not stop working when glucose is low. It stops doing some things and starts doing others. It responds more strongly to immediate rewards and pays less attention to long-term prospects.

¶ The discoveries about glucose help explain why dieting is a uniquely difficult test of self-control — and why even people with phenomenally strong willpower in the rest of their lives can have such a hard time losing weight. They start out the day with virtuous intentions, resisting croissants at breakfast and dessert at lunch, but each act of resistance further lowers their willpower. As their willpower weakens late in the day, they need to replenish it. But to resupply that energy, they need to give the body glucose. They’re trapped in a nutritional catch-22:

¶ 1. In order not to eat, a dieter needs willpower.

¶ 2. In order to have willpower, a dieter needs to eat.

¶ As the body uses up glucose, it looks for a quick way to replenish the fuel, leading to a craving for sugar. After performing a lab task requiring self-control, people tend to eat more candy but not other kinds of snacks, like salty, fatty potato chips. The mere expectation of having to exert self-control makes people hunger for sweets. A similar effect helps explain why many women yearn for chocolate and other sugary treats just before menstruation: their bodies are seeking a quick replacement as glucose levels fluctuate. A sugar-filled snack or drink will provide a quick improvement in self-control (that’s why it’s convenient to use in experiments), but it’s just a temporary solution. The problem is that what we identify as sugar doesn’t help as much over the course of the day as the steadier supply of glucose we would get from eating proteins and other more nutritious foods.

¶ The benefits of glucose were unmistakable in the study of the Israeli parole board. In midmorning, usually a little before 10:30, the parole board would take a break, and the judges would be served a sandwich and a piece of fruit. The prisoners who appeared just before the break had only about a 20 percent chance of getting parole, but the ones appearing right after had around a 65 percent chance. The odds dropped again as the morning wore on, and prisoners really didn’t want to appear just before lunch: the chance of getting parole at that time was only 10 percent. After lunch it soared up to 60 percent, but only briefly. Remember that Jewish Israeli prisoner who appeared at 3:10 p.m. and was denied parole from his sentence for assault? He had the misfortune of being the sixth case heard after lunch. But another Jewish Israeli prisoner serving the same sentence for the same crime was lucky enough to appear at 1:27 p.m., the first case after lunch, and he was rewarded with parole. It must have seemed to him like a fine example of the justice system at work, but it probably had more to do with the judge’s glucose levels.

¶ It’s simple enough to imagine reforms for the parole board in Israel — like, say, restricting each judge’s shift to half a day, preferably in the morning, interspersed with frequent breaks for food and rest. But it’s not so obvious what to do with the decision fatigue affecting the rest of society. Even if we could all afford to work half-days, we would still end up depleting our willpower all day long, as Baumeister and his colleagues found when they went into the field in Würzburg in central Germany. The psychologists gave preprogrammed BlackBerrys to more than 200 people going about their daily routines for a week. The phones went off at random intervals, prompting the people to report whether they were currently experiencing some sort of desire or had recently felt a desire. The painstaking study, led by Wilhelm Hofmann, then at the University of Würzburg, collected more than 10,000 momentary reports from morning until midnight.

¶ Desire turned out to be the norm, not the exception. Half the people were feeling some desire when their phones went off — to snack, to goof off, to express their true feelings to their bosses — and another quarter said they had felt a desire in the past half-hour. Many of these desires were ones that the men and women were trying to resist, and the more willpower people expended, the more likely they became to yield to the next temptation that came along. When faced with a new desire that produced some I-want-to-but-I-really-shouldn’t sort of inner conflict, they gave in more readily if they had already fended off earlier temptations, particularly if the new temptation came soon after a previously reported one.

¶ The results suggested that people spend between three and four hours a day resisting desire. Put another way, if you tapped four or five people at any random moment of the day, one of them would be using willpower to resist a desire. The most commonly resisted desires in the phone study were the urges to eat and sleep, followed by the urge for leisure, like taking a break from work by doing a puzzle or playing a game instead of writing a memo. Sexual urges were next on the list of most-resisted desires, a little ahead of urges for other kinds of interactions, like checking Facebook. To ward off temptation, people reported using various strategies. The most popular was to look for a distraction or to undertake a new activity, although sometimes they tried suppressing it directly or simply toughing their way through it. Their success was decidedly mixed. They were pretty good at avoiding sleep, sex and the urge to spend money, but not so good at resisting the lure of television or the Web or the general temptation to relax instead of work.

¶ We have no way of knowing how much our ancestors exercised self-control in the days before BlackBerrys and social psychologists, but it seems likely that many of them were under less ego-depleting strain. When there were fewer decisions, there was less decision fatigue. Today we feel overwhelmed because there are so many choices. Your body may have dutifully reported to work on time, but your mind can escape at any instant. A typical computer user looks at more than three dozen Web sites a day and gets fatigued by the continual decision making — whether to keep working on a project, check out TMZ, follow a link to YouTube or buy something on Amazon. You can do enough damage in a 10-minute online shopping spree to wreck your budget for the rest of the year.

¶ The cumulative effect of these temptations and decisions isn’t intuitively obvious. Virtually no one has a gut-level sense of just how tiring it is to decide. Big decisions, small decisions, they all add up. Choosing what to have for breakfast, where to go on vacation, whom to hire, how much to spend — these all deplete willpower, and there’s no telltale symptom of when that willpower is low. It’s not like getting winded or hitting the wall during a marathon. Ego depletion manifests itself not as one feeling but rather as a propensity to experience everything more intensely. When the brain’s regulatory powers weaken, frustrations seem more irritating than usual. Impulses to eat, drink, spend and say stupid things feel more powerful (and alcohol causes self-control to decline further). Like those dogs in the experiment, ego-depleted humans become more likely to get into needless fights over turf. In making decisions, they take illogical shortcuts and tend to favor short-term gains and delayed costs. Like the depleted parole judges, they become inclined to take the safer, easier option even when that option hurts someone else.

¶ “Good decision making is not a trait of the person, in the sense that it’s always there,” Baumeister says. “It’s a state that fluctuates.” His studies show that people with the best self-control are the ones who structure their lives so as to conserve willpower. They don’t schedule endless back-to-back meetings. They avoid temptations like all-you-can-eat buffets, and they establish habits that eliminate the mental effort of making choices. Instead of deciding every morning whether or not to force themselves to exercise, they set up regular appointments to work out with a friend. Instead of counting on willpower to remain robust all day, they conserve it so that it’s available for emergencies and important decisions.

¶ “Even the wisest people won’t make good choices when they’re not rested and their glucose is low,” Baumeister points out. That’s why the truly wise don’t restructure the company at 4 p.m. They don’t make major commitments during the cocktail hour. And if a decision must be made late in the day, they know not to do it on an empty stomach. “The best decision makers,” Baumeister says, “are the ones who know when not to trust themselves.”

1. Object Relational Mappers

ORMs are popular among developers but not among performance experts.  Why is that?  Primarily these two engineers experience a web application from entirely different perspectives.  One is building functionality, delivering features, and results are measured on fitting business requirements.  Performance and scalability are often low priorities at this stage.  ORMs allow developers to be much more productive, abstracting away the SQL difficulties of interacting with the backend datastore, and allowing them to concentrate on building the features and functionality.

On the performance side the picture is a bit different.  By leaving SQL query writing to an ORM, you are faced with complex queries that the database cannot optimize well.  What's more ORMs don't allow easy tweaking of queries, slowing down the tuning process further.

 

2. Synchronous, Serial, Coupled or Locking Processes

Locking in a web application operates something like traffic lights in the real world.  Replacing a traffic light with a traffic circle often speeds up traffic dramatically.  That's because when you're out somewhere in the country where there's very little traffic, no one is waiting idly at a traffic light for no reason.  What's more even when there's a lot of traffic, a traffic circle keeps things flowing.  If you need locking, better to use InnoDB tables as they offer granular row level locking than table level locking like MyISAM tables.

Avoid things like semi-synchronous replication that will wait for a message from another node before allowing the code to continue.  Such waits can add up in a highly transactional web application with many thousands of concurrent sessions.

Avoid any type of two-phase commit mechanism that we see in clustered databases quite often.  Multi-phase commit provides a serialization point so that multiple nodes can agree on what data looks like, but they are toxic to scalability.  Better to use technologies that employ an eventually consistent algorithm.

3. One Copy of Your Database

Without replication, you rely on only one copy of your database.  In this configuration, you limit all of your webservers to using a single backend datastore, which becomes a funnel or bottleneck.  It's like a highway that is under construction, forcing all the cars to squeeze into one lane.  It's sure to slow things down.  Better to build parallel roads to start with, and allow the application aka the drivers to choose alternate routes as their schedule and itinerary dictate.

4. Having No Metrics

Having no metrics in place is toxic to scalability because you can visualize what is happening on your systems.  Without this visual cue, it is hard to get business units, developers and operations teams all on the same bandwagon about scalability issues.  If teams are having trouble groking this, realize that these tools simple provide analytics for infrastructure.

There are tons of solutions too, that use SNMP and are non-invasive.  Consider Cacti, Munin, OpenNMS, Ganglia and Zabbix to name a few.  Metrics collections can involve business metrics like user registrations, accounts or widgets sold.  And of course they should also include low level system cpu, memory, disk & network usage as well as database level activity like buffer pool, transaction log, locking sorting, temp table and queries per second activity.

5. Lack of Feature Flags

Applications built without feature flags make it much more difficult to degrade gracefully.  If your site gets bombarded by a spike in web traffic and you aren't magically able to scale and expand capacity, having inbuilt feature flags gives the operations team a way to dial down the load on the servers without the site going down.   This can buy you time while you scale your webservers and/or database tier or even retrofit your application to allow multiple read & write databases.

Without these switches in place, you limit scalability and availability.

WebAppSec/Secure Coding Guidelines

Home » WebAppSec » Secure_Coding_Guidelines

From MozillaWiki

< WebAppSec

Jump to: navigation, search

Content

• 1 Introduction
• 2 Status
• 3 Layout
• 4 Easy Quick Wins
• 5 Secure Coding Guidelines
  • 5.1 Authentication
    • 5.1.1 Password Complexity
      • 5.1.1.1 Critical Sites
    • 5.1.2 Password Rotation
    • 5.1.3 Account Lockout and Failed Login
    • 5.1.4 Password Reset Functions
    • 5.1.5 Email Change and Verification Functions
    • 5.1.6 Password Storage
      • 5.1.6.1 Old Password Hashes
      • 5.1.6.2 Migration
  • 5.2 Session Management
    • 5.2.1 Session ID Length
    • 5.2.2 Session ID Creation
    • 5.2.3 Inactivity Time Out
    • 5.2.4 Secure Flag
    • 5.2.5 HTTP-Only Flag
    • 5.2.6 Logout
  • 5.3 Access Control
    • 5.3.1 Presentation Layer
    • 5.3.2 Business Layer
    • 5.3.3 Data Layer
  • 5.4 Input Validation
    • 5.4.1 Goal of Input Validation
    • 5.4.2 JavaScript vs Server Side Validation
    • 5.4.3 Positive Approach
    • 5.4.4 Robust Use of Input Validation
    • 5.4.5 Validating Rich User Content
  • 5.5 Output Encoding
    • 5.5.1 Preventing XSS
    • 5.5.2 Preventing SQL Injection
    • 5.5.3 Preventing OS Injection
    • 5.5.4 Preventing XML Injection
  • 5.6 Cross Domain
    • 5.6.1 Preventing CSRF
    • 5.6.2 Preventing Malicious Site Framing (ClickJacking)
    • 5.6.3 3rd Party Scripts
    • 5.6.4 Connecting with Twitter, Facebook, etc
  • 5.7 Secure Transmission
    • 5.7.1 When To Use SSL/TLS
    • 5.7.2 Don't Allow HTTP Access to Secure Pages
    • 5.7.3 Implement STS
  • 5.8 Content Security Policy (CSP)
  • 5.9 Logging
  • 5.10 Admin Login Pages
  • 5.11 Uploads
    • 5.11.1 General Uploads
    • 5.11.2 Image Upload
    • 5.11.3 Archive Uploads
  • 5.12 Error Handling
    • 5.12.1 User Facing Error Messages
    • 5.12.2 Debug Mode
    • 5.12.3 Formatting Error Messages
    • 5.12.4 Recommended Error Handling Design
• 6 Further Reading
• 7 Contributors

Introduction

The purpose of this page is to establish a concise and consistent approach to secure application development of Mozilla web applications and web services. The information provided here will be focused towards web based applications; however, the concepts can be universally applied to applications to implement sound security controls and design.

This page will largely focus on secure guidelines and may provide example code at a later time.

Status

The secure coding guidelines page is a living document and constantly updated to reflect new recommendations and techniques. Information that is listed is accurate and can be immediately used to bolster security in your application. If you have comments, suggestions or concerns please email mcoates <at> mozilla.com

Layout

The guidelines are discussed within logical security areas. Instead of discussing how to prevent each and every type of attack, we focus on a secure approach to designing an application. Within each section there is a listing of the types of the attacks these controls are geared to protect against. However, this document is not intended to serve as an in-depth analysis of the attack types, rather a guide to creating a secure application.

Easy Quick Wins

Here are a few items that are often missed and are relevant for most every website.

• For all cookies set the HTTPOnly and Secure flag
• Make sure login pages are only served on HTTPS and all authenticated pages are only served on HTTPS
• Don't trust any user data (input, headers, cookies etc). Make sure to validate it before using it

Secure Coding Guidelines

Authentication

Attacks of Concern

• online & offline brute force password guessing
• user enumeration
• mass account lockout (Account DoS)
• offline hash cracking (time trade-off)
• lost passwords

Password Complexity

All sites should have the following base password policy:

• Passwords must be 8 characters or greater
• Passwords must require letters and numbers
• Blacklisted passwords should be implemented (contact infrasec for the list)

Critical Sites

Examples: addons.mozilla.org, bugzilla.mozilla.org, or other critical sites.

Critical sites should add the following requirements to the password policy:

• Besides the base policy, passwords should also require at least one or more special characters.

Password Rotation

Password rotations have proven to be a little tricky and this should only be used if there is lack of monitoring with-in the applications and there is a mitigating reason to use rotations. Reasons being short password, or lack of password controls.

• Privileged accounts - Password for privileged accounts should be rotated every: 90 to 120 days.
• General User Account - It is also recommended to implement password rotations for general users if possible.
• Log Entry - an application log entry for this event should be generated.

Account Lockout and Failed Login

Account Lockouts vs login failures should be evaluated based on the application. In either case, the application should be able to determine if the password being used is the same one over and over, or a different password being used which would indicate an attack.

The error message for both cases should be generic such as:

Invalid login attempts (for any reason) should return the generic error message

The username or password you entered is not valid

Logging will be critical for these events as they will feed up into our security event system and we can then take action based on these events. The application should also take action. Example would be in the case that the user is being attacked, the application should stop and/or slow down that user progress by either presenting a captcha or by doing a time delay for that IP address. Captcha's should be used in all cases when a limit of failed attempts has been reached.

Password Reset Functions

The password reset page will accept the username and then send an email with a password reset link to the stored email address for that account.

The following message should be returned to the user regardless if the username or email address is valid:

An email has been sent to the requested account with further information. If
 you do not receive an email then please confirm you have entered the same   
 email address used during account registration.

We do not want to provide any information that would allow an attacker to determine if an entered username/email address is valid or invalid. Otherwise an attacker could enumerate valid accounts for phishing attacks or brute force attack.

Email Change and Verification Functions

Email verification links should not provide the user with an authenticated session.

Email verification codes must expire after the first use or expire after 8 hours if not used.

Password Storage

Separate from the password policy, we should have the following standards when it comes to storing passwords:

• Passwords stored in a database should using the hmac+bcrypt function.

The purpose of hmac and bcrypt storage is as follows:

• bcrypt provides a hashing mechanism which can be configured to consume sufficient time to prevent brute forcing of hash values even with many computers
• bcrypt can be easily adjusted at any time to increase the amount of work and thus provide protection against more powerful systems
• The nonce for the hmac value is designed to be stored on the file system and not in the databases storing the password hashes. In the event of a compromise of hash values due to SQL injection, the nonce will still be an unknown value since it would not be compromised from the file system. This significantly increases the complexity of brute forcing the compromised hashes considering both bcrypt and a large unknown nonce value
• The hmac operation is simply used as a secondary defense in the event there is a design weakness with bcrypt that could leak information about the password or aid an attacker

A sample of this code is here: https://github.com/fwenzel/django-sha2

Old Password Hashes

• Password hashes older than a year should be deleted from the system.
• After a password hash migration, old hashes should be removed within 3 months if user has yet to login for the conversion process.

Migration

The following process can be used to migrate an application that is using a different hashing algorithm than the standard hash listed above. The benefits of this approach is that it instantly upgrades all hashes to the strong, recommended hashing algorithm and it does not require user's to reset their passwords.

Migration Process
Migrate all password hashes entries in the database as follows. This is a one time, offline migration.

Stored in databases in form: {algo}${salt}${migration_hash}

* {algo} is {sha512+MD5},
   * {salt} is a salt unique per-user,
   * {migration_hash} is SHA512(salt + existingPasswordHash)

New hash process for new accounts or password changes
Use standard hashing process [above]

New Login Process
1. Attempt to login user with migration hash. This involves performing the old password hash procedure then adding the salt and finally performing the sha512.

Example: Old password hash process is md5
Migration Hash = sha512(perUserSalt + md5(user supplied password))

2. If authentication via migration hash is successful:

- Use the user's provided password and calculate the New Hash per the algorithm defined above.

- Overwrite the Migration Hash with the New Hash

3. If authentication via migration hash is NOT successful:

- The user may already be on the New Hash. Attempt to directly authenticate using the new hash. If this fails, then the password provided by the user is wrong.

Session Management

Attacks of Concern: Session Hijacking, Session Fixation, Brute Forcing Valid Session IDs

Session ID Length

Session tokens should be 128-bit or greater

Session ID Creation

The session tokens should be handled by the web server if possible or generated via a cryptographically secure random number generator.

Inactivity Time Out

Authenticated sessions should timeout after determined period of inactivity - 15 minutes is recommended

Secure Flag

The "Secure" flag should be set during every set-cookie. This will instruct the browser to never send the cookie over HTTP. The purpose of this flag is to prevent the accidental exposure of a cookie value if a user follows an HTTP link.

Code Examples

HTTP-Only Flag

The "HTTP-Only" flag should be set to disable malicious script access to the session ID (e.g. XSS)

Logout

Upon logout the session ID should be invalidated on the server side and deleted on the client via expiration/overwriting the value.

Access Control

Attacks of Concern Enumeration of site features for targeted attacks, Execution of unauthorized functionality, View or modify unauthorized data

Presentation Layer

Display Features and Functions Granted to User

It is recommended to not display links or functionality that is not accessible to a user. The purpose is to minimize unnecessary access controls messages and minimize privileged information from being unnecessarily provided to users.

Business Layer

Check Access Control Before Performing Action

Ensure that an access control check is performed before an action is executed within the system. A user could craft a custom GET or POST message to attempt to execute unauthorized functionality.

Data Layer

Check Access Control with Consideration of Targeted Data

Ensure that an access control check also verifies that the user is authorized to act upon the target data. Do not assume that a user authorized to perform action X is able to necessarily perform this action on all data sets.

Input Validation

Attacks of Concern: Introduction of Dirty/Malformed Data

Goal of Input Validation

Input validation is performed to minimize malformed data from entering the system. Input Validation is NOT the primary method of preventing XSS, SQL Injection. These are covered in output encoding below.

Input Validation Must Be:

• Applied to all user controlled data
• Define the types of characters that can be accepted (often U+0020 to U+007E, though most special characters could be removed and control characters are almost never needed)
• Defines a minimum and maximum length for the data (e.g. {1,25} )

Examples of Good Input Validation Approaches For each field define the types of acceptable characters and an acceptable number of characters for the input

• Username: Letters, numbers, certain special characters, 3 to 10 characters
• Firstname: Letters, single apostrophe, dash, 1 to 30 characters
• Simple US Zipcode: Numbers, 5 characters

Note: These are just examples to illustrate the idea of whitelist input validation. You'll need to adjust based on the type of input you expect.

JavaScript vs Server Side Validation

Be aware that any JavaScript input validation can be bypassed by an attacker that disables JavaScript or uses a Web Proxy. Ensure that any input validation performed by JavaScript is also performed server side as well.

Positive Approach

The variations of attacks are enormous. Use regular expressions to define what is good and then deny the input if anything else is received. In other words, we want to use the approach "Accept Known Good" instead of "Reject Known Bad"

Example A field accepts a username. A good regex would be to verify 
 that the data consists of the following [0-9a-zA-Z]{3,10}. The data 
 is rejected if it doesn't match.

A bad approach would be to build a list of malicious strings and then 
 just verify that the username does not contain the bad string. This 
 approach begs the question, did you think of all possible bad strings?

Robust Use of Input Validation

All data received from the user should be treated as malicious and verified before using within the application. This includes the following

• Form data
• URL parameters
• Hidden fields
• Cookie data
• HTTP Headers
• Essentially anything in the HTTP request

Validating Rich User Content

It is very difficult to validate rich content submitted by a user. Consider more formal approaches such as HTML Purifier (PHP) or AntiSamy or bleach (Python)

Output Encoding

Output encoding is the primary method of preventing XSS and injection attacks. Input validation helps minimize the introduction of malformed data, but it is a secondary control.

Attacks of Concern: Cross Site Scripting, SQL/OS/LDAP/XML Injection

Preventing XSS

• All user data controlled must be encoded when returned in the html page to prevent the execution of malicious data (e.g. XSS). For example <script> would be returned as &lt;script&gt;
• The type of encoding is specific to the context of the page where the user controlled data is inserted. For example, HTML entity encoding is appropriate for data placed into the HTML body. However, user data placed into a script would need JavaScript specific output encoding

Detailed information on XSS prevention here: OWASP XSS Prevention Cheat Sheet

Preventing SQL Injection

• String concatenation to build any part of a SQL statement with user controlled data creates a SQL injection vulnerability.
• Parameterized queries are a guaranteed approach to prevent SQL injection.
• It's not realistic to always know if a piece of data is user controlled, therefore parameterized queries should be used whenever a method/function accepts data and uses this data as part of the SQL statement.

Further Reading: SQL Injection Prevention Cheat Sheet

Preventing OS Injection

• Avoid sending user controlled data to the OS as much as possible
• Ensure that a robust escaping routine is in place to prevent the user from adding additional characters that can be executed by the OS ( e.g. user appends | to the malicious data and then executes another OS command). Remember to use a positive approach when constructing escaping routinges. Example

Further Reading: Reviewing Code for OS Injection

Preventing XML Injection

• Same approach as OS injection. In addition to the existing input validation, define a positive approach which escapes/encodes characters that can be interpreted as xml. At a minimum this includes the following: < > " ' &

• If accepting raw XML then more robust validation is necessary. This can be complex. Please contact the infrastructure security team for additional discussion

Cross Domain

Attacks of Concern: Cross Site Request Forgery (CSRF), Malicious Framing (Clickjacking), 3rd Party Scripts, Insecure Interaction with 3rd party sites

Preventing CSRF

An attacker creates a self posting form or image tag which executes an action on behalf of the authenticated user. Read more about this attack type here

• Any state changing operation requires a secure random token (e.g CSRF token) to prevent against CSRF attacks
• Characteristics of a CSRF Token
  • Unique per user & per user session
  • Tied to a single user session
  • Large random value
  • Generated by a cryptographically secure random number generator
• The CSRF token is added as a hidden field for forms or within the URL if the state changing operation occurs via a GET
• The server rejects the requested action if the CSRF token fails validation

Note: Some frameworks (such as django) provide this capability. Use the established CSRF protection from the framework instead of creating your own.

Preventing Malicious Site Framing (ClickJacking)

A newer attack that uses page layering and framing to convince the user to click or enter data on particular parts of the screen. These actions are actually sent to the framed site to perform actions unbeknown to the victim user. Read more about this attack type here

Set the x-frame-options header for all responses containing HTML content. The possible values are "DENY" or "SAMEORIGIN".

• DENY will block any site (regardless of domain) from framing the content.
• SAMEORIGIN will block all sites from framing the content, except sites within

the same domain.

The "DENY" setting is recommended unless a specific need has been identified for framing.

Code Examples

3rd Party Scripts

• Careful consideration should be used when using third party scripts. While I am sure everybody would do an initial review, updates to scripts should be reviewed with the same due diligence.
• Ensure any scripts that are used are hosted locally and not dynamically referenced from a third party site.

Connecting with Twitter, Facebook, etc

• If using OAuth make sure the entire chain of communication is over HTTPS. This includes the initial OAuth request and any URLs passed as parameters.
• If redirecting to a login page for the app itself, ensure that URL is HTTPS and also that the selected URL does not simply redirect to a HTTP version
• Ensure the "tweet this" or "like this" button does not generate a request to the 3rd party site simply by loading the Mozilla webpage the button is on (e.g. no requests to third party site without user's intent via clicking on the button)

Secure Transmission

Attacks of Concern: Man in the middle, password theft, session id theft

When To Use SSL/TLS

• All points from the login page to the logout page must be served over HTTPS.
• Ensure that the page where a user completes the login form is accessed over HTTPS. This is in addition to POST'ing the form over HTTPS.
• All authenticated pages must be served over HTTPS. This includes css, scripts, images. Failure to do so creates a vector for man in the middle attack and also causes the browser to display a mixed SSL warning message.

Don't Allow HTTP Access to Secure Pages

• Never provide an authenticated page or a login page over HTTP. HTTPS should be used for the login landing page and all subsequent authenticated pages.
• The most secure approach is to display a warning when a user requests the HTTP page to instruct the user to bookmark or type the HTTPS page for future use. However, the more common approach is to just redirect from the HTTP request to the HTTPS equivalent page.

More info on SSL/TLS design can be found here

Implement STS

Where possible, we should utilize STS headers.

Content Security Policy (CSP)

Develop sites without inline JavaScript so adoption of CSP is easier

https://developer.mozilla.org/en/Introducing_Content_Security_Policy

Logging

See Security/Users_and_Logs

Admin Login Pages

The following are generally blockers for any website using an admin page:

1. Controls are in place to prevent brute force attacks
Options (any of these are fine):

• Admin page behind ssl vpn (most popular option)
• Account Lockout
• CAPTCHA's after 5 failed logins
• IP restrictions for access to the admin page

2. The login page and all admin pages are exclusively accessed over HTTPS. Any attempts to access a HTTP page redirect to HTTPS

3. The session id uses the SECURE flag

4. The session id uses the HTTPOnly flag

Configuring Worpress Admin Pages Securely

Uploads

Attacks of Concern: Malformed user uploads containing JavaScript, HTML or other executable code, Arbitrary file overwrite

General Uploads

Upload Verification

• Use input validation to ensure the uploaded filename uses an expected extension type
• Ensure the uploaded file is not larger than a defined maximum file size

Upload Storage

• Use a new filename to store the file on the OS. Do not use any user controlled text for this filename or for the temporary filename.
• Store all user uploaded files on a separate domain (e.g. mozillafiles.net vs mozilla.org). Archives should be analyzed for malicious content (anti-malware, static analysis, etc)

Public Serving of Uploaded Content

• Ensure the image is served with the correct content-type (e.g. image/jpeg, application/x-xpinstall)

Beware of "special" files

• The upload feature should be using a whitelist approach to only allow specific file types and extensions. However, it is important to be aware of the following file types that, if allowed, could result in security vulnerabilities.

• "crossdomain.xml" allows cross-domain data loading in Flash, Java and Silverlight. If permitted on sites with authentication this can permit cross-domain data theft and CSRF attacks. Note this can get pretty complicated depending on the specific plugin version in question, so its best to just prohibit files named "crossdomain.xml" or "clientaccesspolicy.xml".

• ".htaccess" and ".htpasswd" provides server configuration options on a per-directory basis, and should not be permitted. See http://en.wikipedia.org/wiki/Htaccess

Image Upload

Upload Verification

• Use image rewriting libraries to verify the image is valid and to strip away extraneous content.
• Set the extension of the stored image to be a valid image extension based on the detected content type of the image from image processing (e.g. do not just trust the header from the upload).
• Ensure the detected content type of the image is within a list of defined image types (jpg, png, etc)

Archive Uploads

Upload Verification

• Ensure that the decompressed size of each file within the archive is not larger than a defined maximum size
• Ensure that an uploaded archive matches the type expected (e.g. zip, rar, gzip, etc)
• For structured uploads such as an add-on, ensure that the hierarchy within the archive contains the required files

Error Handling

Attacks of Concern: Sensitive Information Disclosure, System Information Disclosure, Aiding exploitation of other vulnerabilities

User Facing Error Messages

Error messages displayed to the user should not contain system, diagnostic or debug information.

Debug Mode

Debug mode is supported by many applications and frameworks and is acceptable for Mozilla applications. However, debug mode should only be enabled in stage.

Formatting Error Messages

Error messages are often logged to text files or files viewed within a web browser.

• text based log files: Ensure any newline characters (%0A%0C) are appropriately handled to prevent log forging
• web based log files: Ensure any logged html characters are appropriately encoded to prevent XSS when viewing logs

Recommended Error Handling Design

• Log necessary error data to a system log file
• Display a generic error message to the user
• If necessary provide an error code to the user which maps to the error data in the logfile. A user reporting an error can provide this code to help diagnose the issue

Further Reading

• OWASP Top 10
• OWASP Cheat Sheets
• OWASP Guide Project
• Php Sec Library
• Django Security
• Ruby Security

Contributors

Michael Coates - mcoates [at] mozilla.com
Chris Lyon - clyon [at] mozilla.com
Mark Goodwin - mgoodwin [at] mozilla.com

Retrieved from "https://wiki.mozilla.org/WebAppSec/Secure_Coding_Guidelines"