The Failed Promise of Big Data for IT Security

[I started thinking about applying Big Data technology to security-related data when I was first asked to write security guidance for a proof-of-concept Hadoop implementation.  Nothing ever came of my efforts.  The nut was too hard for me to crack with the limited data sets that I had and only a beginner’s understanding of R.  Then security product vendors started coming along claiming that their data analytics engines were going to turn the world upside down for security operations teams.  That never happened either.  I ran into a few articles recently that delved into the dismal record of security analytics and I’ve tried to capture their essence, along with a few thoughts of my own, in this post.]

Big Data systems, or, more correctly, data analytics techniques applied in an IT security context to Big Data-style repositories of log and sensor data, promised to transform IT security by giving organizations and IT security teams deep and automatic insights into malicious behavior.  Vendors touted data analytics for intrusion detection, insider threat activity detection, malware behavior detection, and phishing prevention.  Data analytics techniques (advanced statistical analysis, data mining, machine learning, natural language processing, and so on) would reveal insights that manual methods were simply unable to produce.

This promise remains unfulfilled.  Some theorists have begun to argue that fundamental limitations of the data set itself will prevent the highest hopes of security analytics from EVER being realized.  How can that be?  And what should IT security teams be doing with security analytics products?

In an IT security context, it’s not so much the data management aspects of Big Data solutions that we care about but the analytical methods we can apply to our collected security data.  The purpose of data analytics, from a general perspective, is to achieve some form of insight by extracting interesting or meaningful patterns from large and perhaps dissimilar data sets.  Security analytics applies data analytics methods to security-relevant data in order to assist in.identifying the fingerprint of bad actors, malware, and malicious insiders so that incident response plans can be triggered and further action can be initiated.

What exactly do we mean by data analytics?  Data analytics, in this context, means applying methods such as statistical analysis, data mining, machine learning, and natural language processing to computer system and application logs along with security sensor data (primarily network traffic sensors)  in order to detect and identity improper behavior.  Despite products having being available for years (mostly in the SIEM and DLP spaces), the promises of security analytics are still largely just promises.

“The noise about security analytics has grown deafening in the industry, but operational reality still lags far behind.”

- Gartner, 27 Mar 2017

In the cybersecurity space, the boundary between normal and anomalous behavior isn’t always obvious.  There are specific challenges and limitations inherent in the available data the impact the applicability and accuracy of data analytics techniques.

Some specific challenges in using data analytics with cybersecurity data include:

• Data set availability – There are few reference data sets available for things like insider attacks.  Many patterns that current tools look for are only theoretical.  Attack patterns available on the internet are often either sanitized or only applicable to a given enterprise’s IT fabric.
• Asymmetrical costs for errors – Depending on the use case, mistaken categorization can have a disproportionate cost.  For example, in phishing detection a legitimate email being classified as a phishing attack is an annoyance but has little cost.  Most mail filtering products allow filtered emails to be easily viewed and released.  However, allowing even a single actual phishing attempt through can have significant consequences, should the targeted user release the malicious payload.
• Active adversary – Most data analytics activities are applied against a stream of data whose characteristics are relatively constant and where observing the data doesn’t effect the data generator.  In the cybersecurity space however, malicious adversaries are constantly modifying and upgrading their techniques.  They know their footprint is being scrutinized.  They actively camouflage attacks (e.g. polymorphic viruses) and try new methods when old ones fail.  This adversarial learning means that the value of training data sets for machine learning, for example, will degrade quickly.
• Dynamic and complex environments - Data analytics methods rely heavily on ‘normal’ activity happening in regular repeating patterns.  Known-good business processes happen over and over in regular, knowable patterns which make anomalous behavior stand out, right?  If only our IT environments behaved that way.  IT environments are messy, constantly changing, noisy, fraught with one-time events, and almost always are poorly inventoried (despite what the guy who runs your CMDB tells you).  Virtual technologies, mobile devices, and cloud services have aggravated the situation.  Servers and services will be spun up, run for a bit, then vanish, never to be seen again.  Was that a new production feature or malware?  Only a long and tedious investigation MIGHT tell you whether it was one or the other.
• Base rate fallacy – Base rate fallacy is a formal logical fallacy that occurs with detecting low probability events.  Adversarial attacks are by nature low density, where there are thousands or hundreds of thousands of legitimate transactions to each handful of actual attacks. The nature of trying to classify these low density events will lead to false positives greatly outnumbering actual positives.  This can be frustrating for security staff who investigate alert after alert without finding an actual attack.
• Attack time scales – The time scale of malicious activity varies widely.  Attacks can take place in seconds or a patient adversary might deliberately slow his pace to allow an attack to proceed over the course of days or weeks.  Analysis methods dependent on time ranges can fail depending on the attacker’s mode of operations.

Given these challenges, it’s not surprising that security analytics products have largely failed to deliver the value we were promised.  It’s not that they can’t deliver value.  It’s more that the marketing hype, given the reality of the nature of detecting security events, has never been realistic.

So what’s an IT security professional to do?  They key right now is to focus on specific use cases and avoid general purpose solutions that try to boil the ocean.  Set manageable, measurable, and specific detection targets then use metrics to gauge (and demonstrate) your progress.  Make an active effort not to overwhelm your follow-up team with false positives.  Finally, make sure that you, and your boss, understand that using data analytics isn’t the security panacea that many security product and managed service vendors would have you think that it is.

Security is a process, not a product, and no amount of vendor promises will make your environment either more compliant or more secure.  Just buying tools and implementing them without an understanding of what you’re trying to accomplish will just add to the noise, not increase the signal.  Improving the security of your IT environment, using security analytics or any other technology for that matter, requires time, resource, and manpower commitment and should be driven by your use cases and your security framework, not by having tools for the sake of having tools.


For further reading, I recommend:

• “Security Analytics: Essential Data Analytics Knowledge for Cybersecurity Professionals and Students”, Verma et al, IEEE Computing Edge, May 2016
• Gartner Research Note, “Demystifying Security Analytics: Sources, Methods and Use Cases”, 27 Mar 2017
• Gartner Research Note, “Solution Path for Implementing Threat Detection and Incident Response”, 7 Jan 2019

For more information on particular subjects:

• For a good explanation of base rate fallacy, take a look at the Wikipedia page at https://en.wikipedia.org/wiki/Base_rate_fallacy 
• For more on AI and machine learning subjects like Bayesian methods and deep learning, see “Making AI More Human” in the June 2017 issue of Scientific American.  The article includes a discussion of applying machine learning techniques to spam filtering.

The Chicken Tax

Ever wonder why you see TV commercial after TV commercial for pickups when light trucks make up only about a sixth of the vehicles on the road? Ever wonder why pickups seem to be so expensive as compared to cars and why there are far fewer pickup truck models to choose from as compared to car models? As with any economic phenomenon, there are lots of reasons but a big one is the leftover tariffs from a long forgotten trade war that filled the headlines in the early 1960s. That trade war was primarily between the US and Western Europe and became known as the Chicken War.

Prior to the 1950s, chicken wasn't anywhere near the staple food that it is today. Chicken was expensive. The Hoover political slogan "A chicken in every pot" was a promise of luxury for all. Chicken farming methods advanced rapidly in the post-WWII years and soon the U.S. dominated the world chicken market. Cheap U.S. chicken exports particularly hit small Western European farmers hardest and their governments responded with tariffs on imported American chicken. The U.S. responded with tariffs of their own. One such tariff was aimed at West Germany's Volkswagen, particularly the incredibly popular VW Bus (ever wonder why they vanished from the roads?).

Over the intervening years, virtually all of the tariffs from the Chicken War have been repealed except for the U.S. import tax on light trucks and pickups. Car companies have moved to circumvent the tariff, to one extent or another, by either manufacturing their trucks in North America or at least doing final assembly here. Some cargo vehicles are even manufactured overseas as passenger vehicles, brought to the U.S. and then their seats are ripped out and cargo beds installed (which is still cheaper than the tariff).

Protected from competition, some economists argue, light trucks have become a huge profit center for U.S. car companies and their incentive to develop new models and keep prices down has been curtailed... all because of cheap chicken.

Securing Big Data Systems

Big Data is one of the buzzes in the cybersecurity space, both in terms of using Big Data solutions for improving overall IT security as well as securing Big Data implementations.  IT security needs to provide guidance to the applications teams that are implementing Big Data solutions so that these new applications are implemented with security built in from the beginning, rather than trying to bolt on security later.  Many Big Data products haven't had security on the top of their development list as the products rapidly evolve so it's up to users to make sure that these products gets implemented and used correctly.

What is IT security's role in Big Data solution implementation?  In the context of Big Data, the security teams's primary deliverables are (1) security guidance for the Big Data solution architecture and (2) direction for implementing existing security controls and tools into a new technology environment.  What the security team, particularly the security architecture function, needs to do is to use existing patterns and guidance as a baseline and generate draft guidance based on those patterns, on research, and on vendor input.  That guidance should be updated based on feedback from the functional teams and initial informal audits.  As your guidance cycles through the various teams, it will firm up and will eventually become concrete enough to add to your security standards.

The Cloud Security Alliance (www.cloudsecurityalliance.org) Big Data Working Group has does great baseline work related to Big Data and security.  If you're responsible for securing a Big Data implementation, reading their "Expanded Top Ten Big Data Security and Privacy Challenges" is a must.read.

Consider a generic Big Data solution ecosystem:

You can map your security concerns using the CSA taxonomy.  They map as:

When you apply these concerns to the generic ecosystem, you get something that looks like:

This mapping can give you a jumping off point for any security guidance document that you develop.  Apply your set of basic security practices (e.g. RBAC, centralized authentication, encryption, server standards, etc.) and categorize them according to ecosystem components and the CSA top ten and your guidance will have a pre-built skeleton that will be easy to flesh out for your specific tools and solution.

I Have A (Cooking) Confession

Confessions of a Reformed Cheese Snob

I was a parmesan cheese snob.  There, I said it.  And not for just any parmesan, mind you, my cheese had to be Parmigiano-Reggiano and had to be aged at least 24 months.  Nothing else would do.  Ever.  Under any circumstances.  God forbid that I subject my completely unrefined palate to anything lesser.  If I was at an Italian restaurant and wanted to put cheese on my pasta, I could feel the cringe run up my spine as I sprinkled on whatever unworthy white stuff they had in the cheese shaker.

Until recently.

A couple of months ago, I decided to up my pasta game.  I sometimes made fresh pasta but my pasta definitely wasn't great.  Even my best efforts turned out to be just okay.  The only thing making it worth the effort was that even mediocre fresh pasta (like mine) beats dried pasta any day, but my work definitely wasn't memorable by any standard.

I wasn't sure what to do until I thought back to an episode of 'Chef's Table' where Massimo Bottura brought in a little old lady from the Modena neighborhood where his restaurant is to teach his chefs how to make pasta.  If anyone knows how to make pasta, it's little old Italian ladies who have been making pasta every day for the last sixty years, right?  They're the quiet Masters of the Art.  And where do you find little old Italian ladies making pasta?  Why, YouTube, of course.  YouTube, literally, has videos of anything and everything.  God bless the internet.

Sure enough, after a few false starts, I found 'Pasta Grannies'.  'Pasta Grannies' is just what it sounds like; it's videos of Italian grannies making pasta.  Vicky Bennison, the creator of the channel, is a British lady who tools around Italy making short and very watchable videos of Italian women, many in their eighties and nineties, who make pasta that I could only dream of making.  I watched a bunch of videos, went to Sam's Italian Market to get some specialty ingredients, and began to cook.

One thing I noticed was that the Pasta Grannies rarely used Parmigiano-Reggiano.  They used local fresh cheeses to a great extent and when they needed a dry cheese, they frequently turned to Pecorino Romano.  What were they doing to that beautiful pasta, using cheese unworthy of their handiwork?  How could these wonderful cooks be committing such a cardinal sin?

I wanted to substitute.  I really did.  But I also wanted to be faithful to their recipes, at least at first.  I told myself that I could "fix" them later.  Then I started cooking.

God, was their pasta good.  And the cheese flavor in the cappelletti and the agnolotti... it was like a salty, cheesy explosion in my mouth.  Holy crap, I realized, these ladies were on to something.  If you were in my kitchen, you probably would have seen the proverbial light bulb go on above my head.

Of course these women were on to something.  Any one of these ladies has forgotten more about good cooking than I know  (and the more I learn, the more I realize how little I actually know).  With my epiphany, my preconceived notions about Parmigiano-Reggiano were shattered forever.  Gone was my cheese snobbery in a puff of 00 flour.

What I realized, yet again, was that there was a place for every ingredient.  For pasta in a heavy sauce with lots of strong flavors, I needed the sharper Pecorino Romano to cut through all that and add its note to the dish.  Parmigiano-Reggiano, with it's more delicate and nutty flavor profile, would get drowned out in a sea of tomato and garlic.  Everything was about balance and the needs of the dish.  I didn't stir fry in extra virgin olive oil.  I didn't make beef stew with filet mignon.  Ingredients had their places, their roles in the opera.  There were places for the stars singing their arias but there were equally important places for the wide array of singers in the chorus.  A delicate soloist would get lost in the volume of the chorus but will shine on center stage.

I still love Parmigiano-Reggiano.  To me, nothing beats the rind of old parmesan cheese, with it's intense, aged flavor.  If I'm shaving cheese on top of a salad or roasted asparagus, you can bet I'll have my block out, aged at least 24 months, of course.  But in my cheese drawer these days is also a block of Pecorino Romano, waiting for its call to the stage.

The 1st Minnesota at Gettysburg

Given the upcoming Memorial Day weekend, I thought I'd share one of the many incredible stories of valor from Gettysburg.  If you've watched the movie 'Gettysburg' or if you've watched Ken Burns' 'Civil War' documentary series (which I highly recommend) then you know the big stories from Gettysburg... Pickett's Charge, the Wheatfield, Little Round Top, and so on... but for every big story there are dozens of little stories.  For this one, I chose the 1st Minnesota.

The 1st Minnesota Volunteer Infantry Regiment was made up of officers and men who had joined in 1861 and had seen action at First Bull Run, the Peninsula Campaign, Antietam, and Fredericksburg.  On the morning of 2 July 1863 the regiment mustered 262 hardened veterans under command of Col. William Colvill.  They were in reserve as part of the II Corps when the Confederate attack fell onto Dan Sickle's exposed III Corps and crushed it in battles we still remember for their ferocity and savagery; the Wheatfield, the Peach Orchard, Devil's Den, and so on.

Gen. Winfield Scott Hancock, commander of the II Corps, was on top of Cemetery Ridge as the Confederates broke through the shattered III Corps.  Advancing rapidly on Cemetery Ridge was Wilcox's Alabama Brigade, some 1,500-1,800 men.  Nothing stood between the Alabama men and the rear of the Union Army, nothing except for the 1st Minnesota.

Hancock knew what had to be done.  He rode to Col. Colvill and ordered his men to attack.  Hancock needed time to find and bring up more troops or the Union line would be split in two and Gettysburg would turn into yet another Southern victory.  The Minnesotans looked behind them at the unprotected supply wagons, field hospitals, and mess tents then looked in front of them at the 1,500-plus advancing rebels.  The 1st Minnesota were not green troops.  They had seen the killing fields of Northern Virginia.  They knew both what needed to be done and what the cost would be... and they attacked.  After the war, Lt. William Lochren would remember, "“Every man realized in an instant what that order meant – death or wounds to us all, the sacrifice of the regiment to gain a few minutes time and save the position.  And every man saw and accepted the necessity for the sacrifice."

Wilcox's Alabamans were stunned by the sudden ferocity of their charge.  The Alabamans were forced to give battle, unequal as it was.  Wilcox would later write, "Three several times did this last of the enemy's lines attempt to drive my men back, and were as often repulsed. This struggle at the foot of the hill on which were the enemy's batteries, though so unequal, was continued for some thirty minutes."

Those thirty minutes, paid for in blood by the men of the 1st Minnesota, gave Hancock enough time to bring up additional troops and force the Alabamans to retreat.  When the smoke cleared, only forty seven Minnesotans were still standing.  All of the senior officers had been killed or wounded and the surviving men were under command of Captain Nathan Messick.  Depending on which historical records you believe, between 215 and 240 Minnesotans were killed or wounded.  No other unit in the history of the United States Armed Forces has lost a higher percentage of its men in a single battle.

As if that wasn't enough story for a lifetime...  Hancock gathered the remnants of the 1st Minnesota, the survivors of the day, plus 86 men who had been previously sent to other parts of the Union line.  He organized them and placed them at the center of his II Corps where he thought they would be safe.  That spot turned out to be the center of the Union line where Pickett's charge would land on July 3rd.

Other men might have broken and run.  No one would begrudge the 1st Minnesota if they did.  They did not.  They fought the 28th Virginia hand-to-hand, capturing their regimental colors.  Captain Messick was killed in the fighting, as was their next commander, Captain Farrell.  When the fighting was done on July 3rd, seventeen additional Minnesota men had been killed or wounded.  For their actions on the 3rd, two soldiers of the 1st Minnesota, Corporal Henry D. O’Brien and Private Marshall Sherman would both receive the Medal of Honor.

In an ironic postscript, in the 1990s a group of Virginian descendants of the 28th Virginia asked for their regimental battle flag back.  The Minnesota Historical Society, who still have that flag today, politely refused.  When the Virginia group threatened to sue, the Attorney General of Minnesota stepped in and essentially said that if anyone wanted that flag back, they would need to go through the State of Minnesota to get it.  The request was quietly withdrawn.

History is Changing

History is changing... quite literally.  I don't mean that the way we view history is changing or the way historians do research is changing, I mean that history itself is changing, and it's because of the internet.  The internet is changing the way that historians work, the availability of resources, who can do really good history, and... consequently... it's changing history itself.

A very wise man used to tell me that there is no such thing as objective history.  Being the naive and intellectually arrogant teenager that I was, I vehemently disagreed with him.  Surely there was an objective series of events that happened in the past and if we just looked hard enough, we could figure out exactly what those events were and capture forever a perfect, and objective, view of exactly what happened at a certain time or a certain place.  I had a Newtonian view of history.  If we could account for all of the people and sequences, we could state, beyond a shadow of a doubt, exactly what happened.  Objective history, right?

Unfortunately for my teenage worldview, the reality is that we don't... and can't... know everything perfectly.  Heck, we don't even know exactly what words Lincoln spoke in his Gettysburg Address and the Gettysburg Address is one of the most famous speeches in American history!   (There are five known manuscripts that vary from version to version and there are additional versions from contemporary accounts, including an AP reporter's account who wrote it down in shorthand as he listened.)  How can we hope to know what happened in a conversation between two generals at some obscure battle or what was said between two congressmen as they struck a back room deal?  Beyond the events themselves, trying to determine an individual's emotional state, motivations, fears, or other deeply personal feelings is effectively impossible.  What happened here or there and, more importantly, why it happened, is sometimes not much more than guesswork.  That's where historians and the tools of their trade come into play and that's where the internet is fundamentally changing everything.

I recently read "John Bell Hood: The Rise, Fall, and Resurrection of a Confederate General" by Stephen Hood.  I had seen "Sam" Hood speak at several Civil War conferences (thank you, C-SPAN) and I found his talks fascinating.  Sam Hood is a indirect descendant of General John Bell Hood and isn't a "professional" historian.  He's just a smart guy with a love of history who happens to be a meticulous researcher and a pretty good writer.  He's produced a well-researched book about General Hood that's as much an exercise in historiography (the study of historical writings) as it is a history of the life of John Bell Hood.  General Hood had been widely trashed by historians for the last 50 years (to the point where he got trashed in Ken Burns' "Civil War").  Sam Hood thought it seemed a bit unfair and when he started digging, it turned out that it was.

Sam Hood took General Hood's biographers to task and rightly so.  Sam Hood went back through the footnotes in these scholarly biographies, dug back to original sources, and tried to ferret out what really happened in certain situations.  It turned out that, for whatever reason, mid-20th century historians engaged in hyperbole, speculation, and some flat out making up crap when it came to General Hood.  Time and again, Sam Hood was able to cite primary sources that showed biographer bias and how they ignored source material counter to their conclusions.  Sam Hood also turned a harsh light on the intellectual laziness of later 20th century historian who simply repeated, and in some cases, embellished, the unfair or untrue earlier stories about General Hood.  To some extent, the internet made this renewed examination of General Hood possible.

One advantage Sam Hood had was the internet and the vast sources that have been digitized and put online by numerous libraries and projects.  That doesn't excuse earlier historians for not checking primary sources or picking and choosing only only primary sources that suited their story line.  The biographies cited by Sam Hood were largely written by professional historians, history professors at prestigious universities, and at least part of their job was to check their sources vigorously.  It certainly didn't excuse the second wave of historians who simply repeated the stories from the first without doing the additional research of verifying claims.  It reminds me of one brand of false news cycle where a fringe web site states something as fact, two other fringe web sites quote the first, then a mainstream site posts it, claiming that they have an original source and two additional sources that have verified whatever ridiculous claim was originally made.

I was still brooding over Sam Hood's book when I searched YouTube for lectures on General Hood.  Sure enough, there was a recent lecture by one of the Hood biographers that Sam Hood had taken to task.  Sam Hood hadn't ripped him too badly except on the question of repeating fairly serious charges against General Hood without going back to original sources.  The lecture was recent enough that it would have been after Sam Hood's book was generally available.  So I watched it.

At first, the lecturer (and I don't want to name names) avoided the more controversial parts of Hood's military career, which were mostly towards the end of the Civil War.  When he got to that point in General Hood's career, the speaker alluded briefly, and I thought somewhat rudely, to Sam Hood's book.  Referring to Sam Hood somewhat insultingly as a "shirt tail historian", the lecturer defended his practice of using stories from previous biographers as essentially something that everyone does so it was okay.  The speaker did certainly calm some of his rhetoric towards the events around the fall of Atlanta and take a more neutral position so at least in that sense, Sam Hood's book had had some success.

I was still thinking about the Hood lecture and the "everybody does it" excuse, when I saw that there was an upcoming lecture about the Grant presidency (thank you again, C-SPAN).  I decided to watch it.

The lecture was by Charles Calhoun and was in conjunction with the release of his book "The Presidency of Ulysses S. Grant".  It turned out to be fascinating to watch and, in part, addressed the same kinds of issues that were raised by Sam Hood.  Dr. Calhoun addressed both the history of the Grant administration as well as the historiography around it.  And, it turns out, the opinion of biographers regarding the Grant administration has changed considerably over time.  Early in the 20th century, Grant was ranked as one of the worst presidents (even below James Buchanan) although over the century rose to be somewhere in the middle.  Dr. Calhoun was just as fascinated with this change in perception as he was in the Grant administration itself.

Dr. Calhoun went on to talk about how it took eight years to write the book.  Among the reasons that he cited for taking so long, he spoke about going back to primary sources to make sure he got the story as correct as possible.  He mentioned letters, diaries, newspapers, and other contemporary sources.  There was no "I copied from this guy because that's good enough".  It was, in a way, a remarkable presentation, especially set against the lecture about General Hood that I had watched only a few days before.  Was easier access to primary sources a factor in Dr. Calhoun's decision?  He didn't answer that question specifically but I have to wonder if it was.  Regardless, Dr. Calhoun made it clear that not going back to primary sources does a disservice to future readers and, if an author truly wants accuracy, they have to do that work.

There is no objective history but there's certainly no excuse for sloppy history.  More and more archives are being digitized.  More and more archival material is being translated and being made available in multiple languages.  The past never changes but our lens into it, and the history we see through that lens, is clearer than ever.

Lee was overrated

Robert E. Lee is overrated as a general.  There, I said it. I'm not saying that Lee was a bad general, far from it. There's no doubt that Lee was a great general, one of the finest to take a Civil War army into the field, but to read many histories of the Civil War, you’d think that he was an infallible military genius who was a mix of Hannibal, Napoleon, and Patton, with a dash of Mars himself thrown in for good measure. It's this canonization with which I take issue.

I’ve been reading a lot of Civil War history books lately and Lee as a military genius is often assumed as a given.  He’s the immortal icon of the Lost Cause, the noble Virginian who led a proud and honorable defense of his homeland despite overwhelming odds against the foul and oppressive Union horde.  As the story goes, he won again and again with nothing more than a rag tag army of barefoot country boys facing off against the mighty Union military machine.  It certainly makes for a good story, as long as you don’t let the facts get in the way of a good yarn.

First, Lee’s opposing generals early in the war sucked.  They might have been the worst crew of generals that have ever led major armies on the winning side of a war.  Prior to Gettysburg, Lee faced McClellan, Pope, Burnside, and Hooker.  McClellan was afraid of his own shadow, Pope was so indecisive that it nearly got his entire army killed at Second Manassas, Burnside thought the attack at Fredericksburg was a good idea (across the  Rappahannock against the dug in Confederate troops on Marye’s Heights), and Hooker was more interested in self-promotion than in being a good officer.  It wasn’t until Meade that Lee faced an army with even a passably competent commander, and Meade was just okay (although you could argue that he was actually pretty good, but that's another another story for another day).  When the Union finally found a truly hard-charging commander in the form of Grant, Lee’s winning streak was over for good.

Second, Lee’s subordinate officers, particularly at the beginning of the war, were by and large much better officers than opposing Union subordinate officers.  From the corps commanders to the the division commanders down to the brigadiers and regimental officers, Confederate officers were much more skilled, had better training overall, and were far less political.  The vast majority of southern general officers were military school graduates.  The Union army’s officer corps was stuffed full of political appointees.  On the Confederate side, Lee could lean on Jackson, Longstreet, Stuart, Early, Hood, Ewell, and so on.  The Union got dunces like Dan Sickles, the political general who, without orders, marched III Corps out to the Peach Orchard and the Wheatfield, abandoning the Cemetery Ridge line and Little Round Top on the second day of Gettysburg, and came within a hair of losing the battle for the Union.  This advantage steadily wore down as the war went on, as Confederate officers were killed and Union political generals were sacked or forced into backwater commands.  Certainly by 1864, this particular advantage had largely evaporated, despite the emergence of a few new, solid Confederate generals like Mahone.

Third, at the beginning of the war, Confederate cavalry was vastly superior to Union cavalry. Armies at the time were dependent on cavalry for a wide variety of functions, most notably intelligence gathering and enemy intelligence denial. Brandy Station in 1863 was the first time that Union and Confederate cavalry fought even close to evenly. Stuart stepped up his game a little bit after Brandy Station but the advantage continued to erode as the Union officer corps improved, the organization of Union cavalry forces was improved, attrition worked against the South, and the Union weaponry (e.g. multi-shot carbines) advanced beyond Confederate capabilities.

Even the strategy of Lee’s greatest military campaigns could be disputed as bad ideas.  Tactically, yes, he won a number of victories, but at what cost?  He left a quarter of his army on the field at Antietam and another quarter the next year at Gettysburg.  The South didn’t have those kinds of numbers of men to spare.  The North did and Grant used that grim calculus to his advantage when he put Lee’s army through the meat grinder of the Overland Campaign in 1864.  By the spring of 1865 it wasn’t a question of if the Army of Northern Virginia was going to surrender, only when.  Would Lee have done better by following a Fabian-like strategy, more similar to Washington’s strategy versus the British?  Perhaps. It's an interesting and on-going debate. You can certainly make the argument that it would have avoided the costly invasions of Maryland and Pennsylvania.  However, the political reality was that without recognition from foreign powers, succession was destined to fail, just as the American Revolution might very well have failed without recognition from the French. That being said, there's enough meat on both sides of that particular bone for history students to write doctoral dissertations for decades to come on whether or not Lee's invasions of the North were political necessities.

Lee, especially from a tactical perspective, was a unique and outstanding commander.  He had the initial advantages of incompetent opponents, superior subordinates, and better cavalry but when those advantages eroded, he showed that he was still merely mortal.