I just read 'Gone With The Wind'

 I just finished reading ‘Gone With The Wind’.  Why, I hear you ask, in this day and age, would you read such a horrible piece of racist trash?  My answer is because it’s both a great work of American literature and an important historical narrative.  

For all of the disturbing imagery contained in ‘Gone With The Wind’, there’s no doubt that it’s a beautifully written book.  The images evoked by Margaret Mitchell’s writing are vivid and easily draw the reader back into Scarlett O’Hara’s world.  The characters are rich and complex.  Mitchell’s description of far off historic events from Scarlett’s perspective from the streets of Atlanta or the steps of Tara both capture the impact on Southern society as well as the confusion and anxieties her characters experience as their world is turned upside down.  From that perspective, it’s a masterfully written story.

From another perspective, ‘Gone With The Wind’ is a fascinating illustration of the difference between history and memory.  Margaret Mitchell wasn’t a historian.  She was a storyteller and  journalist who grew up in an affluent part of Atlanta in the early 1900s.  She heard stories about the war from her older relatives as well as surviving veterans.  She was educated, going to well-regarded Atlanta schools before one year at Smith College.  ‘Gone With The Wind’ didn’t spring from the mind of some fringe author.  Mitchell’s portrayal of the South during the Civil War and Reconstruction captures the popular memory held by many, including some historians, in the 1920s.  Her book reflects the early 20th century popular memory of the Civil War and reflects the power and invasiveness of the Lost Cause mythology.

‘Gone With The Wind’ depicts part of how the Lost Cause was born and how it took such firm hold.  For people not familiar with the Lost Cause, it was a historical ideology that held that the Confederacy’s cause was just and honorable, and that those that fought for the Confederacy were heroic.  It essentially ignores the role of slavery in triggering the war, and focusses on states’ rights, unfair tariffs, Northern aggression, and other reasons for why the war began.  Supporting this central thesis are other nonsensical arguments like slaves were happier as slaves, slaveholders were all kind and gentle, the Ku Klux Klan was an honorable bunch of men who were just protecting the honor of the South, and so on.  Quoting from the book, “The Lost Cause was stronger, dearer now in their hearts that it had ever been at the height of its glory.  Everything about it was sacred, the graves of the men who died for it, the battle fields, the torn flags, the crossed sabres in their halls, the fading letters from the front, the veterans.”  It was this Lost Cause philosophy that formed the ideological basis behind the implementation of Jim Crow and justified the two-tiered segregationist society that gripped the Deep South for a century after Appomattox.  ‘Gone With The Wind’, in a way, documents the birth of the Lost Cause, in some ways celebrating it, and lets us see just how this pseudo-history gripped popular memory and formed the basis for so many people accepting white supremacy and segregation as simply how it was supposed to be.

It’s this peek into mainstream memory that fascinates me so much about Mitchell’s book.  ‘Gone With The Wind’ was a huge bestseller and won a Pulitzer Prize, above and beyond being turned into a hugely successful Hollywood blockbuster.  This was the antebellum South that many Americans in the first half of the 20th century looked at and said, “Yep.  That’s the way it was.”  It reminds us of how easy it is for popular memory to be fooled into believing a beautiful myth rather than an ugly reality.

Should you read 'Gone With  The Wind' too?  Maybe.  If you're thin-skinned and the movie horribly offended you, I wouldn't recommend it.  The movie was downright pleasant compared to the book.  The book was astonishingly racist and held absolutely nothing back.  Every trope of the Lost Cause was front and center and shouted from the rooftops.  However, if you can get past that, and want to explore the dichotomy between history and memory, 'Gone With The Wind' is a great case study.

Untwisting Lincoln

Untwisting Lincoln

President Abraham Lincoln, one of the greatest presidents the United States has ever had, gets regularly attacked by neo-Confederates, Confederate apologists, Lost Cause true believers, white supremacists, and a host of conspiracy nut jobs, antigovernment hate groups, and other wack-a-doodles that have somehow convinced themselves that the man who saved our country and ended slavery is somehow the most vicious tyrant to ever walk the land.

Nothing could be further from the truth.

The scholarship that these deluded souls fall back on is sketchy at best and outright lies at worst. One technique that the Lincoln bashers use is to pick and choose their sound bites, carefully selecting snippets and then taking them out of context and using those out-of-context quotes to back whatever contrived motivation that they want to ascribe.

For example, a commonly twisted quotation is:

“My paramount object in this struggle is to save the Union, and is not either to save or to destroy slavery. If I could save the Union without freeing any slave I would do it, and if I could save it by freeing all the slaves I would do it; and if I could save it by freeing some and leaving others alone I would also do that.”

Lincoln bashers frequently trot out this quote to show that Lincoln didn’t give a damn about the slaves and he didn’t give a damn about African Americans. All Lincoln cared about was his “unjust” war to stop the Confederate states from exercising their (non-existent) Constitutional right to secede, blah, blah, blah.

Out of context, you can certainly draw the logical conclusion that Lincoln didn’t give a damn about the slaves. Your conclusion would, however, be incorrect.

First, you have to look at the context of the quote. This quote is from a letter that Abraham Lincoln wrote to Horace Greeley on 22 Aug 1862. This letter was in response to a sweeping abolitionist editorial that Greeley had published entitled, “The Prayer of Twenty Millions”. Greeley was a leader of the Radical Republican movement and the editor of a widely read newspaper in New York. His editorial referenced the millions of people in the North and called for Lincoln to move decisively on the destruction of slavery and complete emancipation.

At the time of the exchange, the Second Confiscation Act had been passed but the Emancipation Proclamation was not yet public, although Lincoln had circulated preliminary drafts among his Cabinet. Lincoln was still trying to find a middle ground and was advocating a gradual elimination of slavery, along with compensation for slaveholders (which had been the British model) as well as possible colonization of emancipated slaves to either the Caribbean or Africa. Lincoln’s fear was that sudden and complete emancipation, which Greeley called for, would cause the border states such as Kentucky and Maryland, to secede as well.

The entire text of Lincoln’s letter is as follows.  It's short and well worth reading.

Executive Mansion, Washington, August 22, 1862.

Hon. Horace Greeley: Dear Sir.

I have just read yours of the 19th. addressed to myself through the New-York Tribune. If there be in it any statements, or assumptions of fact, which I may know to be erroneous, I do not, now and here, controvert them. If there be in it any inferences which I may believe to be falsely drawn, I do not now and here, argue against them. If there be perceptable in it an impatient and dictatorial tone, I waive it in deference to an old friend, whose heart I have always supposed to be right.

As to the policy I "seem to be pursuing" as you say, I have not meant to leave any one in doubt.

I would save the Union. I would save it the shortest way under the Constitution. The sooner the national authority can be restored; the nearer the Union will be "the Union as it was." If there be those who would not save the Union, unless they could at the same time save slavery, I do not agree with them. If there be those who would not save the Union unless they could at the same time destroy slavery, I do not agree with them. My paramount object in this struggle is to save the Union, and is not either to save or to destroy slavery. If I could save the Union without freeing any slave I would do it, and if I could save it by freeing all the slaves I would do it; and if I could save it by freeing some and leaving others alone I would also do that. What I do about slavery, and the colored race, I do because I believe it helps to save the Union; and what I forbear, I forbear because I do not believe it would help to save the Union. I shall do less whenever I shall believe what I am doing hurts the cause, and I shall do more whenever I shall believe doing more will help the cause. I shall try to correct errors when shown to be errors; and I shall adopt new views so fast as they shall appear to be true views.

I have here stated my purpose according to my view of official duty; and I intend no modification of my oft-expressed personal wish that all men every where could be free.

Yours, A. Lincoln.

The key part that gets ignored by the Lincoln Haters Club is the last sentence of the letter. Lincoln’s position was what he thought was legal, what had been passed by Congress, and what he saw as Constitutional.  Lincoln's position was what he thought was appropriate for a President who was trying to save the Union.  Lincoln’s personal feelings were as he had stated on multiple occasions before and would state multiple times after, that he was an abolitionist and he believed in emancipation.

Lincoln wanted emancipation done legally and in such a way that it could not be undone by the courts. Remember, in 1862, the Supreme Court was still under slave-sympathizer control and Chief Justice Roger Taney had been the one that wrote the Dred Scott decision. Taney’s summed up his pro-slavery and profoundly racist position in that decision by stating that blacks were “altogether unfit to associate with the white race either in social or political relations, and so far inferior that they had no rights which the white man was bound to respect, and that the negro might justly and lawfully be reduced to slavery for his benefit.”  Knowing that any legal challenge than ended up getting to the Supreme Court would undoubtedly end up in a pro-slavery ruling, Lincoln had to walk a fine legal line.  Lincoln may have personally wanted emancipation, like Greeley did, but Lincoln was bound by the duties and limitations of his office.

For a much more in-depth treatment of Lincoln's views on slavery, I strongly recommend Eric Foner's outstanding book, "The Fiery Trial: Abraham Lincoln and American Slavery".

Nathan Bedford Forrest Day

Nathan Bedford Forrest Day is a National Embarrassment

July 13th is Nathan Bedford Forrest Day in Tennessee.  Each year, by law, the governor of Tennessee is required to sign a proclamation declaring the anniversary of Forrest’s birth as a “special day of observance”. 

Celebrating the life of Nathan Bedford Forrest is a national disgrace and Tennessee should be ashamed of itself.

Confederate memorials and Civil War remembrances are a hot topic these days.  How do you remember the positive contributions of some men without commemorating their rebellion?  Do you honor Jefferson Davis for being Secretary of War under Franklin Pierce or Robert E. Lee for 32 years of distinguished service to the U.S. Army including being Superintendent of West Point?  How do you reconcile Stonewall Jackson’s record as a Confederate general with the fact that he knowingly broke Virginia law by teaching slaves to read or that in the 1850s he helped start a Sunday school for black children, which he continued to fund with his Confederate army pay?  Most men aren’t all good or all bad, but somewhere in between.  Trying to remember the good without celebrating the bad makes these Confederate remembrances all that much more difficult.  The exception to that problem, however, is Nathan Bedford Forrest.  That man was all bad.

Wait! Is this the same Nathan Bedford Forrest that Shelby Foote so fondly remembered in Ken Burns’ Civil War documentary series?  How can that be?  Didn’t Foote called him one of the authentic geniuses of the Civil War era?

Shelby Foote, along with the state of Tennessee, is wrong.

Don't get me wrong, I love Shelby Foote’s writing.  I’ve read several of his books and they were remarkable pieces of narrative history.  I think Foote’s narrations adds depth and a degree of beauty, for the want of a better term, to Burns’ Civil War (and I think that documentary series ought to be required watching for every high school student in America).  I agree with virtually all of Foote’s observations about the Civil War, about its impact on us as a nation, and on how we remember it.  But where it comes to Nathan Forrest, I vehemently disagree with him.

There is nothing worth celebrating in the life of Nathan Bedford Forrest.  His life was without merit.  He was a hateful man, an unabashed racist and white supremacist, and a man who made two fortunes from the sweat of black men’s brows.  His supposed military prowess is vastly overrated.  Even his supposed enlightenment at the end of his life was nothing more than a sham.

Let’s examine the highlights of Forrest’s life.  He was born poor but became wealthy as a planter, using slave labor, and as a slave trader.  By the time the Civil War broke out, he had become one of the richest men in the South.  He recruited and equipped cavalry units, rising to command his own group of raiders.

Forrest was militarily most successful as a raider.  He was a savage fighter, fearsome in battle, and personally killed many men.  But his military success beyond raiding was limited.  As a cavalry commander as part of the Army of Tennessee, for example, he was an abject failure.  He understood small unit tactics and taking advantage of the ground in front of him but he didn’t understand large scale operations or show any sort of strategic skill.  He didn’t understand the role of cavalry within a larger army context.  His savage nature and callous disregard for human life peaked at the Battle of Fort Pillow, where he and his men massacred several hundred U.S. troops, black and white, after they had already surrendered.

After the war, Forrest continued his racist and white supremacist ways.  He joined the fledgling Ku Klux Klan in 1866 or 1867 and became the Klan’s first national leader and Grand Wizard.  He was a delegate to the 1868 Democratic Convention where the motto of the convention was “Let White Men Rule”. (The 1868 presidential election was easily the most racist in history.  President Trump would have to multiply his rhetoric by about a thousand to even come close to 1868.)

Forrest would persist in supporting white supremacist causes and candidates and resisting Reconstruction into the 1870s.  In the mid-1870s, as his health was failing him, Forrest defenders will claim that he had a change of heart and began to embrace reconciliation.  As proof, they offer up speeches that Forrest made, such as the so-called Pole-Bearer’s Speech.  However, what Forrest said and what he did were two different things.

While Forrest was publicly talking about reconciliation, we’d call it being politically correct today, he was building his fortune on the backs of people who were being treated barely better than slaves.  Forrest started farms that leveraged Mississippi’s new Convict Lease System, a system that would be used during the Jim Crow era to arrest black men on trumped up charges, then lease them out to local landowners for far less than those landowners would have to pay if they had to actually use hired labor.  The Convict Lease System, the chain gangs, would persist in various forms throughout the South until the middle of the twentieth century.

I firmly believe that we need to take a deep breath, slow down, and rationally approach what we want to do about Confederate monuments and commemorations.  We need to have a dialog that engages all of the constituents so that we can find common solutions.  However, I don’t believe that at all about anything to do with Nathan Bedford Forrest.  The Tennessee legislature should revoke their 1921 law.  If the law isn’t revoked, the Governor of Tennessee should refuse to enforce it.  Monuments to Forrest should be removed and melted down.  Let’s end the national embarrassment of commemorating this man.

When in Gettysburg

When In Gettysburg

If you find yourself in Gettysburg, it’s easy to get caught up in the grand stories of clashing armies, of Pickett’s Charge or Little Round Top, in the valor and the bravery of the men who fought and died there.  However, if you look closely in the quieter corners, you can find poignant reminds of the true cause of the Civil War.  That cause was slavery, plain and simple, the evil need of one group of men to build their society on the backs and the labors of others.

On Cemetery Ridge, the location of the main Union defensive line, just a few hundred yards north of the Angle, the so-called High-Water Mark of the Confederacy, is the Abraham Brian Farm.  If you find yourself in Gettysburg, you should stop there and spend a few moments reflecting on the family that lived there and how the Civil War had been at their doorstep for more than a decade before the first shots were fired at Fort Sumter.

Abraham Brian was a free black man who, in 1857, bought a small farm just south of Gettysburg, the bustling small town county seat of Adams County, Pennsylvania.  Despite living in a free state, Abraham worked his land knowing that just 10 miles to the south was the Mason-Dixon line and slavery.  In the 1840s and 1850s, Northern free blacks were routinely kidnapped, dragged south and sold into slavery.  Abraham Brian didn’t need to look any further than his own family to be reminded every day of that sobering reality.

Abraham’s wife, Catherine “Kitty” Payne, mother of two of his children, had passed just a few years before and she was his constant reminder.  Kitty stands as a vivid illustration for all of us that in the mid-1800s, free didn’t always mean free.

Kitty Payne had been born a slave in 1816, property of the Maddox family of Huntly, Virginia.  After Samuel Maddox died in 1837, his widow Mary continued to live for a few years in Virginia before moving to Adams County in 1843.  When Mary moved north, she brought Kitty and her four children with her.  Once settled in Pennsylvania, Mary freed them.  Records are scant on where Kitty lived or what she did during the next two years but presumably she lived a quiet life and raised her family.  That quiet life, however, wouldn’t last long.  Mary Maddox’s nephew, feeling like he had been unfairly stripped of his inheritance, sent bounty hunters north and on July 24, 1845, they kidnapped Kitty and her children, returning them to Virginia and slavery.

Gettysburg was stunned and the kidnapping made local headlines.  Quaker abolitionists took up Kitty's cause and gathered enough money to assist Kitty in her legal fight to remain free.  Kitty ended up being one of the very few victims of southern kidnappers to be fortunate enough to win back her freedom, being declared free in 1846.  Kitty and three of her children (one had died during her captivity) returned to Gettysburg and she eventually married Abraham Brian, having two children with him during their short time together before she passed away in 1851.

When the Army of Northern Virginia invaded the North in 1863, free blacks living in Pennsylvania knew full well the fate that would befall them if the Southern Army caught them.  Some 1,000 free blacks would eventually be rounded up and sent south as part of Lee’s plunder of the state.  Brian and his family fled Gettysburg when they heard of Lee’s invasion, returning after the battle to find their home wrecked and over a hundred soldiers buried on their property.  The farm was never the same.  Brian sold it in 1868, taking a job in Gettysburg and living there until his death in 1879.  Abraham and Kitty are both buried in the Lincoln Cemetery in Gettysburg.

For more information about the Brian farm, see this page on the Stone Sentinels website.

You can read more about Kitty Payne on this National Park Service page.

The Failed Promise of Big Data for IT Security

[I started thinking about applying Big Data technology to security-related data when I was first asked to write security guidance for a proof-of-concept Hadoop implementation.  Nothing ever came of my efforts.  The nut was too hard for me to crack with the limited data sets that I had and only a beginner’s understanding of R.  Then security product vendors started coming along claiming that their data analytics engines were going to turn the world upside down for security operations teams.  That never happened either.  I ran into a few articles recently that delved into the dismal record of security analytics and I’ve tried to capture their essence, along with a few thoughts of my own, in this post.]

Big Data systems, or, more correctly, data analytics techniques applied in an IT security context to Big Data-style repositories of log and sensor data, promised to transform IT security by giving organizations and IT security teams deep and automatic insights into malicious behavior.  Vendors touted data analytics for intrusion detection, insider threat activity detection, malware behavior detection, and phishing prevention.  Data analytics techniques (advanced statistical analysis, data mining, machine learning, natural language processing, and so on) would reveal insights that manual methods were simply unable to produce.

This promise remains unfulfilled.  Some theorists have begun to argue that fundamental limitations of the data set itself will prevent the highest hopes of security analytics from EVER being realized.  How can that be?  And what should IT security teams be doing with security analytics products?

In an IT security context, it’s not so much the data management aspects of Big Data solutions that we care about but the analytical methods we can apply to our collected security data.  The purpose of data analytics, from a general perspective, is to achieve some form of insight by extracting interesting or meaningful patterns from large and perhaps dissimilar data sets.  Security analytics applies data analytics methods to security-relevant data in order to assist in.identifying the fingerprint of bad actors, malware, and malicious insiders so that incident response plans can be triggered and further action can be initiated.

What exactly do we mean by data analytics?  Data analytics, in this context, means applying methods such as statistical analysis, data mining, machine learning, and natural language processing to computer system and application logs along with security sensor data (primarily network traffic sensors)  in order to detect and identity improper behavior.  Despite products having being available for years (mostly in the SIEM and DLP spaces), the promises of security analytics are still largely just promises.

“The noise about security analytics has grown deafening in the industry, but operational reality still lags far behind.”

- Gartner, 27 Mar 2017

In the cybersecurity space, the boundary between normal and anomalous behavior isn’t always obvious.  There are specific challenges and limitations inherent in the available data the impact the applicability and accuracy of data analytics techniques.

Some specific challenges in using data analytics with cybersecurity data include:

• Data set availability – There are few reference data sets available for things like insider attacks.  Many patterns that current tools look for are only theoretical.  Attack patterns available on the internet are often either sanitized or only applicable to a given enterprise’s IT fabric.
• Asymmetrical costs for errors – Depending on the use case, mistaken categorization can have a disproportionate cost.  For example, in phishing detection a legitimate email being classified as a phishing attack is an annoyance but has little cost.  Most mail filtering products allow filtered emails to be easily viewed and released.  However, allowing even a single actual phishing attempt through can have significant consequences, should the targeted user release the malicious payload.
• Active adversary – Most data analytics activities are applied against a stream of data whose characteristics are relatively constant and where observing the data doesn’t effect the data generator.  In the cybersecurity space however, malicious adversaries are constantly modifying and upgrading their techniques.  They know their footprint is being scrutinized.  They actively camouflage attacks (e.g. polymorphic viruses) and try new methods when old ones fail.  This adversarial learning means that the value of training data sets for machine learning, for example, will degrade quickly.
• Dynamic and complex environments - Data analytics methods rely heavily on ‘normal’ activity happening in regular repeating patterns.  Known-good business processes happen over and over in regular, knowable patterns which make anomalous behavior stand out, right?  If only our IT environments behaved that way.  IT environments are messy, constantly changing, noisy, fraught with one-time events, and almost always are poorly inventoried (despite what the guy who runs your CMDB tells you).  Virtual technologies, mobile devices, and cloud services have aggravated the situation.  Servers and services will be spun up, run for a bit, then vanish, never to be seen again.  Was that a new production feature or malware?  Only a long and tedious investigation MIGHT tell you whether it was one or the other.
• Base rate fallacy – Base rate fallacy is a formal logical fallacy that occurs with detecting low probability events.  Adversarial attacks are by nature low density, where there are thousands or hundreds of thousands of legitimate transactions to each handful of actual attacks. The nature of trying to classify these low density events will lead to false positives greatly outnumbering actual positives.  This can be frustrating for security staff who investigate alert after alert without finding an actual attack.
• Attack time scales – The time scale of malicious activity varies widely.  Attacks can take place in seconds or a patient adversary might deliberately slow his pace to allow an attack to proceed over the course of days or weeks.  Analysis methods dependent on time ranges can fail depending on the attacker’s mode of operations.

Given these challenges, it’s not surprising that security analytics products have largely failed to deliver the value we were promised.  It’s not that they can’t deliver value.  It’s more that the marketing hype, given the reality of the nature of detecting security events, has never been realistic.

So what’s an IT security professional to do?  They key right now is to focus on specific use cases and avoid general purpose solutions that try to boil the ocean.  Set manageable, measurable, and specific detection targets then use metrics to gauge (and demonstrate) your progress.  Make an active effort not to overwhelm your follow-up team with false positives.  Finally, make sure that you, and your boss, understand that using data analytics isn’t the security panacea that many security product and managed service vendors would have you think that it is.

Security is a process, not a product, and no amount of vendor promises will make your environment either more compliant or more secure.  Just buying tools and implementing them without an understanding of what you’re trying to accomplish will just add to the noise, not increase the signal.  Improving the security of your IT environment, using security analytics or any other technology for that matter, requires time, resource, and manpower commitment and should be driven by your use cases and your security framework, not by having tools for the sake of having tools.


For further reading, I recommend:

• “Security Analytics: Essential Data Analytics Knowledge for Cybersecurity Professionals and Students”, Verma et al, IEEE Computing Edge, May 2016
• Gartner Research Note, “Demystifying Security Analytics: Sources, Methods and Use Cases”, 27 Mar 2017
• Gartner Research Note, “Solution Path for Implementing Threat Detection and Incident Response”, 7 Jan 2019

For more information on particular subjects:

• For a good explanation of base rate fallacy, take a look at the Wikipedia page at https://en.wikipedia.org/wiki/Base_rate_fallacy 
• For more on AI and machine learning subjects like Bayesian methods and deep learning, see “Making AI More Human” in the June 2017 issue of Scientific American.  The article includes a discussion of applying machine learning techniques to spam filtering.

The Chicken Tax

Ever wonder why you see TV commercial after TV commercial for pickups when light trucks make up only about a sixth of the vehicles on the road? Ever wonder why pickups seem to be so expensive as compared to cars and why there are far fewer pickup truck models to choose from as compared to car models? As with any economic phenomenon, there are lots of reasons but a big one is the leftover tariffs from a long forgotten trade war that filled the headlines in the early 1960s. That trade war was primarily between the US and Western Europe and became known as the Chicken War.

Prior to the 1950s, chicken wasn't anywhere near the staple food that it is today. Chicken was expensive. The Hoover political slogan "A chicken in every pot" was a promise of luxury for all. Chicken farming methods advanced rapidly in the post-WWII years and soon the U.S. dominated the world chicken market. Cheap U.S. chicken exports particularly hit small Western European farmers hardest and their governments responded with tariffs on imported American chicken. The U.S. responded with tariffs of their own. One such tariff was aimed at West Germany's Volkswagen, particularly the incredibly popular VW Bus (ever wonder why they vanished from the roads?).

Over the intervening years, virtually all of the tariffs from the Chicken War have been repealed except for the U.S. import tax on light trucks and pickups. Car companies have moved to circumvent the tariff, to one extent or another, by either manufacturing their trucks in North America or at least doing final assembly here. Some cargo vehicles are even manufactured overseas as passenger vehicles, brought to the U.S. and then their seats are ripped out and cargo beds installed (which is still cheaper than the tariff).

Protected from competition, some economists argue, light trucks have become a huge profit center for U.S. car companies and their incentive to develop new models and keep prices down has been curtailed... all because of cheap chicken.

Securing Big Data Systems

Big Data is one of the buzzes in the cybersecurity space, both in terms of using Big Data solutions for improving overall IT security as well as securing Big Data implementations.  IT security needs to provide guidance to the applications teams that are implementing Big Data solutions so that these new applications are implemented with security built in from the beginning, rather than trying to bolt on security later.  Many Big Data products haven't had security on the top of their development list as the products rapidly evolve so it's up to users to make sure that these products gets implemented and used correctly.

What is IT security's role in Big Data solution implementation?  In the context of Big Data, the security teams's primary deliverables are (1) security guidance for the Big Data solution architecture and (2) direction for implementing existing security controls and tools into a new technology environment.  What the security team, particularly the security architecture function, needs to do is to use existing patterns and guidance as a baseline and generate draft guidance based on those patterns, on research, and on vendor input.  That guidance should be updated based on feedback from the functional teams and initial informal audits.  As your guidance cycles through the various teams, it will firm up and will eventually become concrete enough to add to your security standards.

The Cloud Security Alliance (www.cloudsecurityalliance.org) Big Data Working Group has does great baseline work related to Big Data and security.  If you're responsible for securing a Big Data implementation, reading their "Expanded Top Ten Big Data Security and Privacy Challenges" is a must.read.

Consider a generic Big Data solution ecosystem:

You can map your security concerns using the CSA taxonomy.  They map as:

When you apply these concerns to the generic ecosystem, you get something that looks like:

This mapping can give you a jumping off point for any security guidance document that you develop.  Apply your set of basic security practices (e.g. RBAC, centralized authentication, encryption, server standards, etc.) and categorize them according to ecosystem components and the CSA top ten and your guidance will have a pre-built skeleton that will be easy to flesh out for your specific tools and solution.