Information Theory Reading Group
Jul. 29th, 2009 08:50 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
maradyddShortly after my recent post ragging on Dembski, new reader and mathematician ![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
coheleth wrote me asking for some pointers on learning information theory. I also got an anonymous comment asking for the same thing (hello, anonymous reader!), and who am I to say no when someone wants to talk about math?
However, I know for a fact that I have a lot of really sharp readers, many of whom are way better at this kind of thing than I am. So, you are all cordially invited to participate in a paper-reading and discussion group, right here on this very LJ.
Why information theory? Well, because it's the unsung hero of the modern age. Every form of communication we take for granted today -- telephones, cellphones, radio, the Internet, wifi, Bluetooth, you name it -- has its feet firmly planted in information theory. Information theory also helps us to make sense of the world around us, from DNA to black holes. Understanding information theory will help you to be a better scientist, even if you're not one already. On a very fundamental level, information theory has a lot to say about what we can know, what we can do. It's theoretical math for realists.
I figure we'll start with one of the classics -- Claude Shannon's The Mathematical Theory of Communication, available in several formats from the nice people at Bell Labs -- and work our way forward from there, letting the pace set itself. When I was doing this in grad school, we did a paper a week, reading on our own and meeting to discuss for a couple of hours once weekly. Since this is the internet, I figure discussions might go on for a couple of days, so my initial thinking is a paper every two weeks -- that's a week to read, a couple of days for discussion, then a breather before picking up the next one. But that's all assumption; I don't want to drag anyone away from a good discussion, nor do I want to rush anyone.
If you're curious about the subject but think that you're bad at math, then rejoice -- information theory is, to my mind at least, one of the easiest mathematical disciplines for laypeople to understand. It will help if you understand binary numbers (or, better yet, how bases work generally); a grasp of basic probability (i.e., how to compute the likelihood of a certain number coming up on a dice roll) will also be useful, as will ninth-grade algebra. You will also need to understand that the integral of a function is the area under the curve in the graph of that function (or, in three-space, the volume described by rotating that curve around an axis), and the general idea of summation (including the notion of a convergent series, which is an infinite series whose sum has a limit, i.e., it does not grow without bound). But that's it. Seriously. (You don't have to actually understand how to compute an integral. Hell, calculus was fifteen years ago; I barely remember how to do one myself. I am way overspecialised in discrete math, and underequipped for continuous math.)
If anyone's interested but doesn't feel like they have the prerequisites down, I can post something in the next couple of days to get you up to speed; don't be shy.
Where we'll go from Shannon is anyone's guess, and depends mostly on where the discussion goes. We'll likely end up talking about coding theory and compression (as in, how ZIP files work, and how your cellphone is able to hold a reliable connexion without being clobbered by the thousands of other conversations going over the cell network). But we might also get into cryptography and cryptanalysis, information-theoretic security (as in, cryptosystems that can't be broken even if the attacker has all the computational power in the universe -- a favourite subject ofenochsmiles' and mine), astronomy (radio telescopes are completely dependent on information theory), and computability theory, that latter by way of Gregory Chaitin and algorithmic information theory.
My goal here is to deepen and broaden understanding on all levels. In a meatspace paper-reading group that's difficult, but since the net is distributed, I'm hoping that we can address the curiosity of newbies, experts and self-proclaimed "non-math people" alike. Feel free to invite your non-LJ friends, too. (They might want to get OpenID accounts, to make discussion threads easier, but that's certainly not a requirement.)
So! You've got the link, up there in the fourth paragraph; go forth and read. It's 55 pages, so I'm thinking we might want to start with just the first part (pages 1-19 inclusive). We should definitely follow up with part two; I'm less sanguine about part three, but if there's interest, we'll do it.
I'll kick off discussion next Wednesday with some questions and maybe an observation or three. I'm looking forward to having you join us!
coheleth wrote me asking for some pointers on learning information theory. I also got an anonymous comment asking for the same thing (hello, anonymous reader!), and who am I to say no when someone wants to talk about math?However, I know for a fact that I have a lot of really sharp readers, many of whom are way better at this kind of thing than I am. So, you are all cordially invited to participate in a paper-reading and discussion group, right here on this very LJ.
Why information theory? Well, because it's the unsung hero of the modern age. Every form of communication we take for granted today -- telephones, cellphones, radio, the Internet, wifi, Bluetooth, you name it -- has its feet firmly planted in information theory. Information theory also helps us to make sense of the world around us, from DNA to black holes. Understanding information theory will help you to be a better scientist, even if you're not one already. On a very fundamental level, information theory has a lot to say about what we can know, what we can do. It's theoretical math for realists.
I figure we'll start with one of the classics -- Claude Shannon's The Mathematical Theory of Communication, available in several formats from the nice people at Bell Labs -- and work our way forward from there, letting the pace set itself. When I was doing this in grad school, we did a paper a week, reading on our own and meeting to discuss for a couple of hours once weekly. Since this is the internet, I figure discussions might go on for a couple of days, so my initial thinking is a paper every two weeks -- that's a week to read, a couple of days for discussion, then a breather before picking up the next one. But that's all assumption; I don't want to drag anyone away from a good discussion, nor do I want to rush anyone.
If you're curious about the subject but think that you're bad at math, then rejoice -- information theory is, to my mind at least, one of the easiest mathematical disciplines for laypeople to understand. It will help if you understand binary numbers (or, better yet, how bases work generally); a grasp of basic probability (i.e., how to compute the likelihood of a certain number coming up on a dice roll) will also be useful, as will ninth-grade algebra. You will also need to understand that the integral of a function is the area under the curve in the graph of that function (or, in three-space, the volume described by rotating that curve around an axis), and the general idea of summation (including the notion of a convergent series, which is an infinite series whose sum has a limit, i.e., it does not grow without bound). But that's it. Seriously. (You don't have to actually understand how to compute an integral. Hell, calculus was fifteen years ago; I barely remember how to do one myself. I am way overspecialised in discrete math, and underequipped for continuous math.)
If anyone's interested but doesn't feel like they have the prerequisites down, I can post something in the next couple of days to get you up to speed; don't be shy.
Where we'll go from Shannon is anyone's guess, and depends mostly on where the discussion goes. We'll likely end up talking about coding theory and compression (as in, how ZIP files work, and how your cellphone is able to hold a reliable connexion without being clobbered by the thousands of other conversations going over the cell network). But we might also get into cryptography and cryptanalysis, information-theoretic security (as in, cryptosystems that can't be broken even if the attacker has all the computational power in the universe -- a favourite subject of
enochsmiles' and mine), astronomy (radio telescopes are completely dependent on information theory), and computability theory, that latter by way of Gregory Chaitin and algorithmic information theory.My goal here is to deepen and broaden understanding on all levels. In a meatspace paper-reading group that's difficult, but since the net is distributed, I'm hoping that we can address the curiosity of newbies, experts and self-proclaimed "non-math people" alike. Feel free to invite your non-LJ friends, too. (They might want to get OpenID accounts, to make discussion threads easier, but that's certainly not a requirement.)
So! You've got the link, up there in the fourth paragraph; go forth and read. It's 55 pages, so I'm thinking we might want to start with just the first part (pages 1-19 inclusive). We should definitely follow up with part two; I'm less sanguine about part three, but if there's interest, we'll do it.
I'll kick off discussion next Wednesday with some questions and maybe an observation or three. I'm looking forward to having you join us!
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
(no subject)
Date: 2009-07-29 08:45 am (UTC)(no subject)
Date: 2009-07-29 11:56 am (UTC)(no subject)
Date: 2009-08-02 07:02 am (UTC)(no subject)
Date: 2009-07-29 12:39 pm (UTC)(no subject)
Date: 2009-07-29 12:50 pm (UTC)(no subject)
Date: 2009-07-29 02:29 pm (UTC)Dunno if I'll join this lovely chat about it, at least, not until after I complete this fascinating book.
(no subject)
Date: 2009-07-29 02:32 pm (UTC)(no subject)
Date: 2009-07-29 02:36 pm (UTC)(no subject)
Date: 2009-07-29 05:27 pm (UTC)I'm particularly interesting in papers that relate to info theory for reliable data streaming. I'll see if I can pull out some interesting ones for the group.
(no subject)
Date: 2009-07-29 05:39 pm (UTC)(no subject)
Date: 2009-07-29 03:19 pm (UTC)(no subject)
Date: 2009-07-29 03:56 pm (UTC)(no subject)
Date: 2009-07-29 04:38 pm (UTC)(no subject)
Date: 2009-07-29 06:31 pm (UTC)Ooh, shiny. I've occasionally tried to understand information theory but been discouraged for various reasons. Do you mind if I invite some of my friends, or is the intent to have this take place within your social circle?
(no subject)
Date: 2009-07-29 06:32 pm (UTC)(no subject)
Date: 2009-07-29 07:33 pm (UTC)(no subject)
Date: 2009-07-29 07:37 pm (UTC)(no subject)
Date: 2009-07-29 09:07 pm (UTC)(no subject)
Date: 2009-07-30 03:02 am (UTC)I should actually learn more of the continuous stuff for work (real probability theory, stochastic processes, relevant bits of measure theory, and the like), but the discrete stuff is fun.
(no subject)
Date: 2009-07-30 03:14 am (UTC)(no subject)
Date: 2009-08-03 03:34 am (UTC)(no subject)
Date: 2009-08-04 08:58 pm (UTC)(no subject)
Date: 2009-08-04 08:56 pm (UTC)(no subject)
Date: 2009-08-04 08:57 pm (UTC)(no subject)
Date: 2009-08-05 02:24 am (UTC)