Bitcoin Forum
November 10, 2024, 06:46:00 AM *
News: Latest Bitcoin Core release: 28.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 [13] 14 »  All
  Print  
Author Topic: About Mt. Gox flaw from a security expert  (Read 34160 times)
BBanzai
Member
**
Offline Offline

Activity: 84
Merit: 10



View Profile
June 25, 2011, 08:09:34 AM
 #241

I have a silly question, DON'T CHEAT, do it by hand on paper....what is 11010110101011001011111001000111 xor'ed with 00101001010100110100000110111000?  This is not a quiz, it is an exercise.
BBanzai
Member
**
Offline Offline

Activity: 84
Merit: 10



View Profile
June 25, 2011, 08:33:26 AM
 #242

Good lord, this damn Bitcoin thing is making me remember what I used to know - I'm serious, Little Mouse, if you are going to be quoting random stuff about RAID but have never known what logic-gates were by building, testing, and debugging them yourself,if you insist on talking about security perimeters...I mean parameters, without having written bad code yourself and been embarrassed enough to fix it BEFORE you handed it in, have never had your own FreeBSD (I always thought OpenBSD was slightly better, but that was 10 years ago) get its ass handed to you from the get-go because your video-card wasn't handled properly by the stable X Server version that was out that year.....you have to fail by trying, and you have to understand that this is how we all learn.  You weren't criticised so much for your ignorance as for your attitude.  Although the pretense of experience was in there too, it was the lesser offense.
kokjo
Legendary
*
Offline Offline

Activity: 1050
Merit: 1000

You are WRONG!


View Profile
June 25, 2011, 11:09:56 AM
 #243

I have a silly question, DON'T CHEAT, do it by hand on paper....what is 11010110101011001011111001000111 xor'ed with 00101001010100110100000110111000?  This is not a quiz, it is an exercise.
alot of ones: 11111....

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
BBanzai
Member
**
Offline Offline

Activity: 84
Merit: 10



View Profile
June 25, 2011, 06:07:00 PM
 #244

Yay!!!!!!  That would either be very true or an exceedingly large number.
jgraham
Full Member
***
Offline Offline

Activity: 140
Merit: 100


<Pretentious and poorly thought out latin phrase>


View Profile
June 25, 2011, 06:23:23 PM
 #245

Running a RaidZ is a plus for security.

Meh, unless you say how (which is kind of the thing that you keep "forgetting" to do here) it's not much of a point.

RAIDZ in terms of it's disk layout isn't significantly different (in function) from a number of other dynamically expandable RAIDs system.  The fact it is part of ZFS means that it has the checksum and copy-on-write facilities which make it a good choice for preventing accidental loss of data integrity. However that's not really what we're talking about when we mean 'security'.   What does ZFS really bring to the security party though?  Assuming all the solaris stuff came with it into FreeBSD - ACLs? What doesn't have at least some ACL support these days? although I'm not positive how the various features trade off between systems e.g. delegation.  For the record I'm running ZFS on my gentoo box.

(For reference the above is closer to an actual discussion on the security merits of RAIDZ as opposed to just calling it "a plus" which is more of a lame pontification).

Quote
Maybe you should go to them and tell that it's not true, linux is as safe as BSD, maybe even better. I'm sure they will be more than happy to follow your, jgraham and the other linux kid advices.

You have (temporarily?) confused yourself...who's advising to switch from FreeBSD to Linux?  Nobody.  Who's arguing that someone who needs a secure environment switch from Linux to FreeBSD - you.   Just to look at the logic.   Assuming I'm right and a properly secured Linux box is as good as a properly secured FreeBSD box (although in each case 'properly secured' would mean different things).   There would be zero advantage in moving (assuming the existing platform is meeting their needs).

I'm rather good with Linux.  If you're having problems with your mining rig I'll help you out remotely for 0.05.  You can also propose a flat-rate for some particular task.  PM me for details.
marvinmartian
Full Member
***
Offline Offline

Activity: 224
Merit: 100



View Profile
June 25, 2011, 06:28:06 PM
 #246

People, this isn't about Linux vs FreeBSD. 

This is about good sysadmins vs mediocre (or even bad) ones.

The latter make life difficult for the entire world, particularly when they manage machines that a) deal with $$$ and b) have a large user base.

Running a RaidZ is a plus for security.

Meh, unless you say how (which is kind of the thing that you keep "forgetting" to do here) it's not much of a point.

RAIDZ in terms of it's disk layout isn't significantly different (in function) from a number of other dynamically expandable RAIDs system.  The fact it is part of ZFS means that it has the checksum and copy-on-write facilities which make it a good choice for preventing accidental loss of data integrity. However that's not really what we're talking about when we mean 'security'.   What does ZFS really bring to the security party though?  Assuming all the solaris stuff came with it into FreeBSD - ACLs? What doesn't have at least some ACL support these days? although I'm not positive how the various features trade off between systems e.g. delegation.  For the record I'm running ZFS on my gentoo box.

(For reference the above is closer to an actual discussion on the security merits of RAIDZ as opposed to just calling it "a plus" which is more of a lame pontification).

Quote
Maybe you should go to them and tell that it's not true, linux is as safe as BSD, maybe even better. I'm sure they will be more than happy to follow your, jgraham and the other linux kid advices.

You have (temporarily?) confused yourself...who's advising to switch from FreeBSD to Linux?  Nobody.  Who's arguing that someone who needs a secure environment switch from Linux to FreeBSD - you.   Just to look at the logic.   Assuming I'm right and a properly secured Linux box is as good as a properly secured FreeBSD box (although in each case 'properly secured' would mean different things).   There would be zero advantage in moving (assuming the existing platform is meeting their needs).

"... and the geeks shall inherit the earth."
BBanzai
Member
**
Offline Offline

Activity: 84
Merit: 10



View Profile
June 25, 2011, 06:36:18 PM
 #247

Bravo.  The devil may be in the details, but the more important part is that Mt Gox was overly optimistic and got lazy - I still think Bitcoins are a toy, but they will be far more than that when non-technical people can expect that the technical people won't let their stored value be secured by a kiss and a promise.
iCEBREAKER
Legendary
*
Offline Offline

Activity: 2156
Merit: 1072


Crypto is the separation of Power and State.


View Profile WWW
June 26, 2011, 06:58:12 PM
 #248

In the CS community, it's well known that BSD is more stable, secure, and the best OS for critical infrastructure, while Linux is more friendly, flexible, and better for hobbyists or businesses that can save money (by hiring cheaper Linux fanboi rather than expensive real computer scientists).

Referring to a commonly known fact, such as the security of BSD vs Linux, is not an argument.

If it were a fact, then you would be able to point to some clear and objective evidence of that right?  (Keep in mind that because you are referring to 'security' as some kind of blanket term you'd be responsible for providing that kind of evidence for the majority of aspects of the term and of course how exactly you know that your set of aspects is the majority).

Nice labeling there mac.  This isn't gainsaying.  I, simply as a IT security professional and the holder of a degree in computer science, have seen no set of well-defined, broadly scoped evidence that BSD is superior in "security" to Linux.  Nor in my conversation with other security professionals or members of the CS community (like my alumni, Usenix attendees)  see any clear consensus as to the superiority of BSD.  I have, certainly met people who make that claim but they always seem to fall down when trying to come up with a general definition of security or if they do they fall down in substantiating it with regard to their favored OS/Platform/Giant Spider.  Ergo it seems reasonable to me to call such a term "complex" furthermore given that even the most secure systems from a theoretical point of view can be entirely undone in implementation (such as EMF side-channel attacks on QKDS) it seems again reasonable to me to call such a system "nuanced".  Given these two facts (using the term correctly here).  I think it is entirely justified to be mistrustful of any and all who consider "security' as an open and shut case for product (or platform or giant spider) X over product (you get the idea) Y.

What do you want from me here guy? The two sentences above tell me to look at your use of the term "well-known" as: your opinion of the opinions of two very large groups of which your sample size is probably so small and poorly randomized it's useless.  Not to mention that even if the majority of those two groups held the opinion you claim it still isn't necessarily meaningful   Computer Science and EECS people do not always have a background in computer security.   Making their opinion anywhere from questionable to useless.   Given the size of the groups and the variance in the population's skill set you could easily be getting the opinion of the least qualified people. I mean would you really rank the opinion of someone's who's focus was in Combinatorics or AI or Queuing Theory as equal or greater than Bruce Schneier or (going old school) D. J. Bernstien when it comes to an application or operating systems "security".  If you don't then how many Combinatoricists, AI researchers or Queuing Theorists make one Bruce or Dan?  

Not to mention it's not hard to find high-profile people in computer security who disagree on "well-known" concepts.

You, BB, and Tux may huff, puff, insult, gainsay, and dissemble until blue in the face, but that won't change anything in reality.

Bickering and playing word games don't cut it, especially when a statistical modeling expert (specialized in computer security) is schooling you on the facts and logic of the issue at hand.  Thanks Maud-dib, for attempting to educate these stubborn script kiddies (1337 RHEL cert notwithstanding, LOL!).

I repeat: Referring to a commonly known fact, such as the security of BSD vs Linux, is not an argument.

Quote
What is the most secure operating system?
In: Operating Systems, Computer Security

http://wiki.answers.com/Q/What_is_the_most_secure_operating_system
   
Answer:

Security is a difficult and sometimes controversial thing to analyze. The only truly "secure" operating systems are those that have no contact with the outside world. The firmware in your DVD player is a good example.

Among all modern general purpose operating systems (Windows, Mac OS X, Linux, Solaris, FreeBSD, NetBSD, OpenBSD) the most secure by default is by far OpenBSD.

OpenBSD has an extremely stringent security auditing policy; only two remote attack vulnerabilities have been found in the last ten years. This is because OpenBSD doesn't create a large attack surface by running a large number of networked apps.


I've met Linus Torvalds in person.  He's a nice guy, and it sucks his baby is being represented here by fanboi suffering from Tiny E-peen Complex.



██████████
█████████████████
██████████████████████
█████████████████████████
████████████████████████████
████
████████████████████████
█████
███████████████████████████
█████
███████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
███████████████████████████
██████
██████████████████████████
█████
███████████████████████████
█████████████
██████████████
████████████████████████████
█████████████████████████
██████████████████████
█████████████████
██████████

Monero
"The difference between bad and well-developed digital cash will determine
whether we have a dictatorship or a real democracy." 
David Chaum 1996
"Fungibility provides privacy as a side effect."  Adam Back 2014
Buy and sell XMR near you
P2P Exchange Network
Buy XMR with fiat
Is Dash a scam?
iCEBREAKER
Legendary
*
Offline Offline

Activity: 2156
Merit: 1072


Crypto is the separation of Power and State.


View Profile WWW
June 26, 2011, 07:14:29 PM
 #249


- BSD has proactive security, Linux security is reactive

- BSD is designed from the ground for security, Linux instead has a more chaotic architecture

Very well put; an elegant statement.

I love that some ITT Tech foolio is questioning the methodology of a trained statistical modeler.

It's entertaining how, when his on-topic nonsense is corralled and put down, he simply disputes whether or not you *really* have an MS in stat. 

The word for when someone contradicts everything you write is called 'gainsaying.'  It's not a polite or nice thing to do, and hence is often considered trolling or flamebaiting.

You've led the donkeys to water, but the stubborn asses won't drink.

No wonder you've added the 3 Stooges to your plonk file (wish BTCforum would add an 'ignore' feature).

It didn't have to be this difficult:

http://tinyurl.com/3lfxm4x
Quote
Is Linux the most secure OS?

Linux-based systems get a lot of press in IT trade publications. A lot of that press relates to its security characteristics. In fact, some claim “Linux is the most secure operating system (OS) of them all.” Such statements are, of course, unsupportable hyperbole; while many Linux distributions may outshine both MS Windows and Apple MacOS X by a significant margin, there’s evidence to suggest that most Linux distributions are not up to the standards of FreeBSD, for instance — let alone OpenBSD, with possibly the best security record of any general-purpose operating system.



██████████
█████████████████
██████████████████████
█████████████████████████
████████████████████████████
████
████████████████████████
█████
███████████████████████████
█████
███████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
███████████████████████████
██████
██████████████████████████
█████
███████████████████████████
█████████████
██████████████
████████████████████████████
█████████████████████████
██████████████████████
█████████████████
██████████

Monero
"The difference between bad and well-developed digital cash will determine
whether we have a dictatorship or a real democracy." 
David Chaum 1996
"Fungibility provides privacy as a side effect."  Adam Back 2014
Buy and sell XMR near you
P2P Exchange Network
Buy XMR with fiat
Is Dash a scam?
kokjo
Legendary
*
Offline Offline

Activity: 1050
Merit: 1000

You are WRONG!


View Profile
June 26, 2011, 07:29:32 PM
 #250

im finding it interesting that you are quoting answers.com are you serious?

"The whole problem with the world is that fools and fanatics are always so certain of themselves and wiser people so full of doubts." -Bertrand Russell
iCEBREAKER
Legendary
*
Offline Offline

Activity: 2156
Merit: 1072


Crypto is the separation of Power and State.


View Profile WWW
June 26, 2011, 09:28:19 PM
 #251

im finding it interesting that you are quoting answers.com are you serious?

Do you understand what this means, Professor Gainsayer?

Quote
OpenBSD has an extremely stringent security auditing policy; only two remote attack vulnerabilities have been found in the last ten years. This is because OpenBSD doesn't create a large attack surface by running a large number of networked apps.


It's called 'Answers.com' for a reason!


██████████
█████████████████
██████████████████████
█████████████████████████
████████████████████████████
████
████████████████████████
█████
███████████████████████████
█████
███████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
███████████████████████████
██████
██████████████████████████
█████
███████████████████████████
█████████████
██████████████
████████████████████████████
█████████████████████████
██████████████████████
█████████████████
██████████

Monero
"The difference between bad and well-developed digital cash will determine
whether we have a dictatorship or a real democracy." 
David Chaum 1996
"Fungibility provides privacy as a side effect."  Adam Back 2014
Buy and sell XMR near you
P2P Exchange Network
Buy XMR with fiat
Is Dash a scam?
jgraham
Full Member
***
Offline Offline

Activity: 140
Merit: 100


<Pretentious and poorly thought out latin phrase>


View Profile
June 27, 2011, 12:59:06 AM
Last edit: June 27, 2011, 02:11:25 PM by jgraham
 #252

You, BB, and Tux may huff, puff, insult, gainsay, and dissemble until blue in the face, but that won't change anything in reality.

True but I'm hardly doing any of those things.  Grin

Quote
Bickering and playing word games don't cut it, especially when a statistical modeling expert (specialized in computer security) is schooling you on the facts and logic of the issue at hand.
What, other than Maud-dibs say-so has you thinking he's any kind of expert in statistics?  I mean other than that he appears to agree with you.  Can you point me to a specific, well supported point he has made?  From where I sit if there was an award for uninformative posts.   I think maud_dib would be a contender.

Quote
I repeat: Referring to a commonly known fact, such as the security of BSD vs Linux, is not an argument.

Actually it is.  We call it an implied argument from popularity.  It's no more compelling than people who say: "It's a well known fact that <racial/ethnic group X> is <deficiency Y>".  

Quote

I think you must be pulling my leg here...any reason that you take the word of this random person on the internet?  I mean other than that they appear to agree with you?   

Quote from: some random person on the internet
Security is a difficult and sometimes controversial thing to analyze. The only truly "secure" operating systems are those that have no contact with the outside world. The firmware in your DVD player is a good example.

This reads a lot like someone who heard a college lecture and is making "broken telephone" mistakes in repeating it.  I've heard this used as a theoretical example.  That is, a system which has no contact with people or machines at all is secure by definition but that's because 'security' is probably being defined as 'Allowing only the right people access to the right things'.  Allowing nobody access to anything is clearly conforming to that definition.  However that is also an example of the most useless system.   So sometimes people use this example to refer to attacks that are network related.  So again, yes removing the ability to talk on the network is yet again, conforming to our definition.    However the network isn't the only way people gain access to information.  Ergo while a non-networked computer is immune to network attacks.   It doesn't mean it's immune from the wrong people getting access to information.   A computer could have no network card but be physically available.   Terminals in our university library were able to access some machines without using a network connection.   As they were hardwired into a serial console connected to the computer.

Using a DVD player as an example is either wrong, dated or unclear.  DVD Players allow physical access and some even allow network access.

Quote from: some random person on the internet
only two remote attack vulnerabilities have been found in the last ten years. This is because OpenBSD doesn't create a large attack surface by running a large number of networked apps.

Actually this isn't quite correct.  The actual tagline was:

"Only two remote holes in the default install, in a heck of a long time!" (emphasis mine)

I addressed this already.  The best argument you can make here is that an OpenBSD box with nothing else installed is secure from a remote attack.  However that doesn't really tell you much about OpenBSD code, review procedure, or their overall security model.   So it doesn't say the average OpenBSD box is secure nor anything about how secure an OpenBSD box would be when running a common application in a production environment and it sure doesn't say anything about when compared to a Linux machine that has been secured by someone qualified to do so. 

What OpenBSD attempts to do is commendable but the statement is closer to marketing hype than a useful security metric.

Quote
I've met Linus Torvalds in person.  He's a nice guy, and it sucks his baby is being reprsented here by fanboi suffering from Tiny E-peen Complex.

Linus has publicly been more critical of the OpenBSD development model than I have been in any of my posts.  In case you keep missing it.  I simply deny that there is clear evidence that in any real-world environment a secured OpenBSD (or a FreeBSD) box is more secure than a secured Linux box.  That isn't saying that OpenBSD isn't good, nor is it saying that Linux is the best.


Quote
Very well put; an elegant statement.
...and inaccurate.  I've already given examples as to how OpenBSD has avoided proactive security measures either because they consider their existing security sufficient or Theo D. has gone a little nuts.

Quote
I love that some ITT Tech foolio is questioning the methodology of a trained statistical modeler.

First, your use of "statistical modeler" instead of simply "statistician" is adorable!  Second I think it's more me saying "Where exactly *is* your methodology?" and maud_dib kind of pretending that methodological transparency isn't important.

Quote from: someone who has certifications but is not CS which Icebreaker keeps implying is a lesser situation
there’s evidence to suggest that most Linux distributions are not up to the standards of FreeBSD, for instance — let alone OpenBSD, with possibly the best security record of any general-purpose operating system.

This article is all over the place.  At first he says there is evidence to suggest but he doesn't say what that is,   The only thing he seems to mention is the OpenBSD tagline (which says nothing about FreeBSD).  He does make this other interesting quote later on:

Quote
One of the most common criteria used by people who don’t really understand security, and by those who do understand it but want to manipulate those who don’t with misdirection and massaged statistics, is vulnerability discovery rates.

Isn't this one of the two primary metrics that maud_dib espoused?  According to this guy he says maud_dib 'doesn't understand security'.   I've already given my rationale for why these rates aren't such a good metric.

He goes on to mention a few more metrics without any rational why these are particularly useful.

Quote
    i) code quality auditing
    ii) default security configuration
    iii) patch quality and response time
    iv) privilege separation architecture

i) I don't agree with this as prima facie it's difficult to express it as a metric.   What units does "code quality auditing" come in?
ii) Likewise this is hard to express usefully as a number and it's really only meaningful to people who are in the habit of deploying systems in their default configs.
iii) If by 'quality' we mean a binary condition consisting of: a) Does it fix the security problem b) Does it cause another security problem.   This is a metric I actually like but there's no information as to how FreeBSD, Linux and OpenBSD differ in this respect.
iv) Again I like this idea but it's difficult to express in units.   Perhaps some categorical?


Quote
It's entertaining how, when his on-topic nonsense is corralled and put down, he simply disputes whether or not you *really* have an MS in stat.

...so on your Internet nobody pretends they're something they're not?  I can see how security seems so easy over there then.  Just make your login sequence ask: "Hey are you *really* supposed to be accessing this system?" since where you are everyone is completely honest.  All attackers will be forced to say "No".  Then you can log them out.

Where I am people regularly attempt to fake it.   As I've already said my position is simple.  Maud_dib has provided very little in the way of what he was attempting to do with his "psi", how it was meaningful to computer security and what data he was supplying to it.  He has had multiple opportunities to clear up some very simple questions.  I think that means the alleged statistician has earned some skepticism.

Besides I've provided a dearth of information as to why I hold the positions I do and I'm open to argument on those points.   So far all you want to say is your position is a "fact" and therefore requires no support.   Which is fine, have any sort of religion about computers you want but it's hardly surprising when those of us above the age of seventeen think the world is a little more complicated than you suggest.

I'm rather good with Linux.  If you're having problems with your mining rig I'll help you out remotely for 0.05.  You can also propose a flat-rate for some particular task.  PM me for details.
iCEBREAKER
Legendary
*
Offline Offline

Activity: 2156
Merit: 1072


Crypto is the separation of Power and State.


View Profile WWW
June 28, 2011, 01:52:38 AM
 #253

So, again, what are the professionals using and why?  And how?

Mt. Gox Uses FreeBSD.

Of course they do. 

It's been the OS of choice for the security-conscious crowd since long before the first generation of cryptocash.

I'd like to see the 3 Stooges who insulted, attacked, and ran you off try to browbeat MtGox into switching to Linux, using similar thug tactics of ganging up and gainsaying everything said to them.

That would be funny!

I wonder why they, who know compsec ever-so-much-better than MtGox, Muad-dib, and myself, simply don't start up their own clearinghouse and compete with MtGox.

MtTux would be 100% Linux, and therefore immune to the security problems presented by BSD (all TWO of them, LOL).





██████████
█████████████████
██████████████████████
█████████████████████████
████████████████████████████
████
████████████████████████
█████
███████████████████████████
█████
███████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
███████████████████████████
██████
██████████████████████████
█████
███████████████████████████
█████████████
██████████████
████████████████████████████
█████████████████████████
██████████████████████
█████████████████
██████████

Monero
"The difference between bad and well-developed digital cash will determine
whether we have a dictatorship or a real democracy." 
David Chaum 1996
"Fungibility provides privacy as a side effect."  Adam Back 2014
Buy and sell XMR near you
P2P Exchange Network
Buy XMR with fiat
Is Dash a scam?
iCEBREAKER
Legendary
*
Offline Offline

Activity: 2156
Merit: 1072


Crypto is the separation of Power and State.


View Profile WWW
June 28, 2011, 02:30:12 AM
 #254

Quote
your use of "statistical modeler" instead of simply "statistician" is adorable!

Adorable?  Thanks.  I guess you have a thing for articulate nerds with a deep understanding of both math and language.

Let's break it down into simple chunks that will be easier for you to digest.

1.  All statistical modelers are statisticians.

2.  Not all statisticians are statistical modelers.


Now just wait a second, and don't get all upset or confused before letting me resolve this seemingly inexplicable paradox of connotation versus denotation.

You see, statistical modeling is part of what's called 'applied statistics.'

To do applied statistics (in the form of modeling) you first need a background in what's called 'theoretical statistics.'


Once you have that, you can stay in the world of theory and be an ivory-tower egghead statistician, or you can enter the real world and help build statistical models of real things (like computer security) by applying your theoretical background, as a statistical modeler.

Some statisticians do leave academia and enter the private sector, but do not build statistical models.  They remain statisticians, and do not become statistical modelers.

Here is some further reading for you, to gratify your demonstrated deep curiosity regarding this mysterious, crucial, and often misunderstood distinction.  Enjoy!

Quote
There are many controversial topics actively discussed among business analysts who follow divergent schools of thought.  The most common schools of thought can be categorized into two groups:  the first group being the theoretical statisticians, and the second being represented by those individuals who embrace “applied” statistics.  Generally, the theoretical statisticians apply what they’ve learned in an academic setting, and follow the “laws” set forth by their institutions.  On the other end of the spectrum, the applied statisticians rely heavily on market testing and key performance indicators (e.g., financial impact) to determine their own set of experientially-based statistical methods and axioms.

Neither school is inherently good or bad.  All seasoned analytic managers have met new analysts who come straight out of school with misconceptions of the value and place for various mathematical procedures and rules.  We’ve all also faced analysts with significant career experience who have carried their academic theoretical statistical knowledge with them as an unchanging edict, despite the limited (or detrimental) applicability of some of these doctrines in the marketplace.  Similarly, we’ve all also encountered business-focused “applied statisticians” whose lack of adherence to theory has resulted in unstable strategic analytic products that look great on paper, but fail in practice.

Of all the points of conflict between theoretical and applied statisticians, one of the most heated relates to the utility of the measurement of colinearity in predictive modeling.  In predictive modeling, colinearity is the amount to which two independent variables correspond to the same dependent variable.  It can also refer to the amount a single independent variable corresponds to a dependent variable.

The theoretical statistician will argue that intensively managing colinearity is of great importance in building predictive models.  A few of the arguments they will cite to support this position include that if colinearity isn’t removed:

    We cannot clearly explain the value of each independent variable in the model’s predictive algorithm
    We are endorsing a final product that may not conform with standard mathematical partiality towards a solution that is parsimonious in nature
    Parameter estimates might be unstable from sample to sample (or from validation to marketplace execution)

The applied statistician will argue that colinearity is not relevant as:

    We are seeking lift , not explanation.  If the new model makes more money in the marketplace, the ability to explain “why” becomes academic
    Parameter estimate stability can be enhanced through various exercises during the model build phase

The reality is that both sides may be correct, at specific application points, and in specific situations.  We just need to moderate academic rigor with real-world findings in order to uncover when to implement a rule, when to bend it, and when to discard it.  To address each of the five points (above):

    Explaining an individual variable’s contribution to a multivariate prediction may or may not have relevance.
        If you are in a market research company, this is a key concern.  You will need to let your clients know not only “what will be,” but “why.”
        If you are in a direct marketing company, explanation may not be relevant.  As an example, if you work for a catalog company, maximum incremental financial lift is far more important than explaining the “percent of predictive value” driven by individual model components.

    Ideally, we want a parsimonious solution as they tend to be more stable.  But, what if you find that your less parsimonious option (having been tested on multiple out-of-time validation samples) is almost identical in stability?  What if, during those same tests you find that it produces a far more robust prediction?  In short:
        Generally, you will want to favor a more parsimonious solution
        But, if you have a model that is relatively less parsimonious, but already proven stable and robust, there may not be any additional value in reworking the solution for the sake of a mathematical preference

    If you are conducting a model building strategy that does not manage colinearity, but is laser-focused on lift, and you find that your parameter estimates are not stable, a likely cause is inadequate sample size in the build data set.  As a result:
        You can increase your sample size substantially (which will typically eliminate this issue)

    For most predictive model applications in industry, lift is the goal.  But you need to be apprised of the perspective of senior management and clients.  Until they are comfortable with your track record, they may require you to explain the nature, source and quantified relevance of each individual variable in your model…and you’ll need to provide this explanation in business terms they can understand

    Managing parameter estimate instability can’t always be achieved:
        The most common way to reduce model instability (caused by collinear variables) is to increase the build and validation sample sizes.  But, for many organizations, there simply isn’t enough data to do this effectively (especially for smaller organizations that are not engaged in direct marketing).
        Another potential parameter estimate instability cure is to examine each variable and appropriately bin them relative to the dependent variable in question.  Keep in mind, though, that the more you bin, the more you will also be reducing variable information value…and this may end up reducing the overall predictive power of the model.

Overall, the positions held by the “pure” theoretical statistician and the “pure” applied statistician both have strengths and weaknesses that can be demonstrated in actual market testing.  To improve effectiveness, each group needs to move beyond a mastery of one philosophy, and become a pragmatist of both.







██████████
█████████████████
██████████████████████
█████████████████████████
████████████████████████████
████
████████████████████████
█████
███████████████████████████
█████
███████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
███████████████████████████
██████
██████████████████████████
█████
███████████████████████████
█████████████
██████████████
████████████████████████████
█████████████████████████
██████████████████████
█████████████████
██████████

Monero
"The difference between bad and well-developed digital cash will determine
whether we have a dictatorship or a real democracy." 
David Chaum 1996
"Fungibility provides privacy as a side effect."  Adam Back 2014
Buy and sell XMR near you
P2P Exchange Network
Buy XMR with fiat
Is Dash a scam?
jgraham
Full Member
***
Offline Offline

Activity: 140
Merit: 100


<Pretentious and poorly thought out latin phrase>


View Profile
June 28, 2011, 02:38:59 AM
Last edit: June 30, 2011, 04:22:22 PM by jgraham
 #255

So, again, what are the professionals using and why?  And how?
If by "professionals" we mean people who are in an industry where security is prioritized.   It's not always clear cut.   The financial institutions I worked for used OS/390 machines simply because they had invested huge amounts of money into them.  Not because of any pretense of security.

I'd like to see the 3 Stooges who insulted, attacked, and ran you off try to browbeat MtGox into switching to Linux, using similar thug tactics of ganging up and gainsaying everything said to them.

I think you have officially entered the twilight zone now.   It was maud_dib who chastised Mt. Gox for using Linux.  Now of course, he says they don't so if we assume, like you have that his opinion was evidence based that is there was something that made him think the OS was at fault (after all he's a "trained statistical modeler" :-)  ) Then either those selfsame indicators would apply to FreeBSD OR the opinion was not evidence based it was assumed based on his presumption that Linux is insecure by comparison.  An opinion neither you nor maud_dib has provided any useful objective evidence for.

But we don't talk about that....just like you don't talk about maud_dibs done more than his fair share of insults...We just talk about the insults he's received.  Am I clear on where you are coming from?

Anyway your implied question has already been answered oh delusional one.  If Linux is as good as BSD then there is little reason to switch.

Quote
I wonder why they, who know compsec ever-so-much-better than MtGox, Muad-dib, and myself, simply don't start up their own clearinghouse and compete with MtGox.

Well, Mt. Gox has made some noobish mistakes but they were all, from my understanding policy and implementation errors.  Unlike maud_dib (initially) I don't have a problem with their choice of OS.  I don't really know anything about starting up a monetary exchange and my side-projects already consume enough of my time.  I really don't see why in your opinion everyone who understands computer security needs to start monetary exchanges but perhaps I'm just not drinking heavily enough.

On the other hand I've already proposed $500 USD in BTC as a prize for a contest for breaking into a hardened Linux box.     The way you talk it would be easy money but considering the way you act I suspect it isn't.  Grin

Maybe you'll answer that now.

Quote
Adorable?  Thanks.  I guess you have a thing for articulate nerds with a deep understanding of both math and language.
No but I'll let you know when you start showing signs of either of those. :-)

Quote
1.  All statistical modelers are statisticians.

2.  Not all statisticians are statistical modelers

Name something from statistics which is not a model of something or a modeling tool.  Grin  If you read your own article you'd see that both the so-called Pure statistician and Applied statistician are modeling something.   The only difference is the kind of validation they are willing to consider.  So the "pure" statistical model is considered valid (in this case) when it conforms to some dogma about colinearity and the "applied" model is valid when (among other things) it succeeds in predicting something.

Anyway you're cute when you just shovel barely applicable google cites and pretend that somehow makes your point.

But in the interests of you actually contributing something....I'll try to keep in mind that when you say "statistical modeler" you mean "Applied statistician".  Not that you show much understanding of what the second term means anyway.

I'm rather good with Linux.  If you're having problems with your mining rig I'll help you out remotely for 0.05.  You can also propose a flat-rate for some particular task.  PM me for details.
BBanzai
Member
**
Offline Offline

Activity: 84
Merit: 10



View Profile
June 29, 2011, 02:09:42 AM
 #256

I already know that I'm a bit of a troll, being a bohunk, backwoods IT guy and all...but the biggest problem I had with math wasn't the math.  It was the conclusion that because your math was brilliant that it must therefore also be true.  The only real things in this world do not just exist in your mind.  What physics do engineers do?
BBanzai
Member
**
Offline Offline

Activity: 84
Merit: 10



View Profile
June 29, 2011, 02:22:12 AM
 #257

(I may be casting my vote for Messr. Graham's arguments, in few words)  Just because it looks good on paper, don't mean it flies.  Doing your homework means trying it yourself, not quoting "authoritative" sources like, oh, Wikipedia.  Don't get me wrong.  I check WP all the time.  And the articles' reference materials.
iCEBREAKER
Legendary
*
Offline Offline

Activity: 2156
Merit: 1072


Crypto is the separation of Power and State.


View Profile WWW
June 30, 2011, 03:21:40 AM
 #258

Quote
1.  All statistical modelers are statisticians.

2.  Not all statisticians are statistical modelers

Name something from statistics which is not a model of something or a modeling tool.  Grin  If you read your own article you'd see that both the so-called Pure statistician and Applied statistician are modeling something.   The only difference is the kind of validation they are willing to consider.  So the "pure" statistical model is considered valid (in this case) when it conforms to some dogma about colinearity and the "applied" model is valid when (among other things) it succeeds in predicting something.

Anyway you're cute when you just shovel barely applicable google cites and pretend that somehow makes your point.

But in the interests of you actually contributing something....I'll try to keep in mind that when you say "statistical modeler" you mean "Applied statistician".  Not that you show much understanding of what the second term means anyway.

Dang it jgraham, stop being so funny or we might end up as friends or something!   Angry

I'm glad you finally accepted that the tension between your overly strict denotation of "statistical modeler" and the common, widespread informal connotation of "applied statistician"  is best resolved, in popular usage (check the job postings for both terms), by the simple understanding that "statistician" implies more theoretical or academic work while "statistical modeler" implies a more applied or industrial frame of reference.

My approach is superior because it preserves linguistic information, while your misinterpretation destroys the sometimes subtle distinction between a working statistician (ie statistical modeler) and the purely theoretical academic egghead (ie capitol-S Statistician).

You seemed to be claiming that 'all statisticians use statistical models, therefore all terms referring to them are all interchangeable, no matter their particular function, specialty, talent, or role.'

That's why I showed you the differences with the copypasta illustrating and demonstrating their existence at length, and in excruciating detail.

I knew that would win you over.   Grin

Cheers brah!

/Hella Laim Flaimwar


██████████
█████████████████
██████████████████████
█████████████████████████
████████████████████████████
████
████████████████████████
█████
███████████████████████████
█████
███████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
████████████████████████████
██████
███████████████████████████
██████
██████████████████████████
█████
███████████████████████████
█████████████
██████████████
████████████████████████████
█████████████████████████
██████████████████████
█████████████████
██████████

Monero
"The difference between bad and well-developed digital cash will determine
whether we have a dictatorship or a real democracy." 
David Chaum 1996
"Fungibility provides privacy as a side effect."  Adam Back 2014
Buy and sell XMR near you
P2P Exchange Network
Buy XMR with fiat
Is Dash a scam?
jgraham
Full Member
***
Offline Offline

Activity: 140
Merit: 100


<Pretentious and poorly thought out latin phrase>


View Profile
June 30, 2011, 02:17:28 PM
Last edit: July 02, 2011, 01:32:45 AM by jgraham
 #259

1.  All statistical modelers are statisticians.

2.  Not all statisticians are statistical modelers
Name something from statistics which is not a model of something or a modeling tool.  Grin  If you read your own article you'd see that both the so-called Pure statistician and Applied statistician are modeling something.   The only difference is the kind of validation they are willing to consider.  So the "pure" statistical model is considered valid (in this case) when it conforms to some dogma about colinearity and the "applied" model is valid when (among other things) it succeeds in predicting something.

Anyway you're cute when you just shovel barely applicable google cites and pretend that somehow makes your point.

But in the interests of you actually contributing something....I'll try to keep in mind that when you say "statistical modeler" you mean "Applied statistician".  Not that you show much understanding of what the second term means anyway.

Quote from: iCEMAKER
I'm glad you finally accepted that the tension between your overly strict denotation of "statistical modeler" and the common, widespread informal connotation of "applied statistician"

Your prejudicial language aside.  I think you misread my post.  I accept that when you use the term 'statistical modeler' you are referring to some idealized trope identified as "applied statistician" by some person on the internet.  For whom the term doesn't really denote a presence or absence of statistical modeling just some polarized ideas about model validation.

Quote from: iCEMAKER
My approach is superior because it preserves linguistic information, while your misinterpretation destroys the sometimes subtle distinction between a working statistician (ie statistical modeler) and the purely theoretical academic egghead (ie capitol-S Statistician).

Not really.  Your approach is simply to assume that you are unquestionably correct for something where 'correct' is difficult to ascertain and without citing any useful corroborating evidence.  Which is just what you did with regard to the opinion of some group about the security of FreeBSD (or perhaps BSD's in general).   It's not much of an approach but I can see how it might fool the locals.

Wheras what I did was just recognize that language is fluid and, your prescriptivism aside allow for your particular definition to stand for the discussion I am having with you.   Back at the office your postings are a subject of much derision by the (few) other mathematicians we employ.  Just sayin...

Quote from: iCEMAKER
That's why I showed you the differences with the copypasta illustrating and demonstrating their existence at length, and in excruciating detail.

In the words of BB's icon...."You keep using those words.  I do not think it means what you think it means."  I think it's kind of obvious that you didn't understand much of what you read.  Since the only thing that was strongly contrasted in your article between these two hypothesized opposites is something you didn't mention and everything else was not directly covered.

"excruciating detail" - I guess, to someone who doesn't understand what they read. - absolutely precious.  If I could keep you like a pet I would.

So now for the second time I accept how you are using the term...any chance you will actually contribute something?  Probably not.

I'm rather good with Linux.  If you're having problems with your mining rig I'll help you out remotely for 0.05.  You can also propose a flat-rate for some particular task.  PM me for details.
BBanzai
Member
**
Offline Offline

Activity: 84
Merit: 10



View Profile
July 05, 2011, 01:00:35 AM
 #260

I will take that as a compliment.  What masks we choose to wear is sometimes as informative as just going around wearing our own faces.
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 [13] 14 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!