Systematic Povery in Education

A while back I found in my hands a list of prices that CBSE books sell for. A curious little dataset, for what could you answer using this information? Well, a simple question one might ask is how expensive is education in the various classes of CBSE? One gets this chart from the data.

 

Almost all books a re cheaper than 250 rupees. Although this chart says nothing about the number of books needed in each class to clear the exams set by CBSE. I remember having to purchase 4-5 books. What about the sum of prices in each class?

 

As expected, 11 and 12 branch out into multiple streams and so have a significantly large number of books, which when multiplied by their prices puts the entire set out of reach for a lot of people.

A man below the poverty line mentioned to be INR 32 a day (IndiaToday) does not earn enough over a year to be able to afford all the books in class 11 and 12. 

Besides school fees / clothing / rent this expense is critical if people must have books to learn. Even if you allow for the segregation of books equally into the main streams of learning (science, humanities, commerce) you are still left with an expense of ~ INR 5000. This is no small amount for someone earning INR 32 a day. It would take almost three months before they can even accumulate this amount.

It would seem that a system which makes access to education difficult in proportion to the need for education will only breed ignorant citizens. Although the CBSE has done tremendous work in making the books this cheap, there are still a ways to go.

If you want to see how these conclusions were reached you can take a look at the code on Github.

Attendance in St. Stephen's College

I've been hunting around for datasets I can relate to for some time now. This fine evening I arrived on the conclusion that I could simply use attendance data from the website of St Stephen's College.

The complete analysis notebook is available as a gist (here's the gist).

Collecting the data was a matter of inspecting how the website was obtaining it's data. A company has been handling our attendance since the beginning it seems. (GreenClouds site link). As far as I'm concerned they've made a mess of the data transfer as I can see. Perhaps it originated in the college itself, perhaps it's due to some mistakes on the company level. Whatever the cause the data format is a mess.

I open up my console and spin up a Python script to get the data from the college website. Takes about two minutes on my Internet connection to collect all the information.

We then proceed to put it in a nice tabular form with columns being Name, percent LA, percent TA, percent PA, admission_year, course.

Things are now ready for graphing.

Graphs

We go on to make Box plots. If you were to line up people in order of their attendance you would get a box plot. The person in the middle of the line is the line cutting the box in the middle. It shows that there are an equal number of people on either side of the line.

You cut each part of the line in half again and you get the bounds of the box.

This one is a gem. What is up with Chem! The attendance of their Lectures is absolutely ridiculous. While everyone more or less has the same attendance in a course Chem people are all over the attendance spectrum! Then again, their Tutorials box plot is neat too!

Yes. I see it too. Math people are consistently in class with a few people as outliers. Nobody else has this kind of spread. 

Perhaps a better way of seeing this would be violin plots where you get to see the density of the people instead of simple boxes. The thickness of the violin denotes how many people people are at that point.

For example we can see that a lot of the math  people are high attendance junkies and a substantial amount of CHE, PHY, PCH and, PCS are low attendance lovers.

There! The average PHI student absolutely does not give a hoot about attendance! The PCH and PCS programs do every class together as is apparent from their spreads. ENG and HST do compete with PHI but PHI does take the cake.

Let's move on to the simplest of all graphs. The Histogram. No explanation needed here.

Looking at the histograms we can see that almost everyone attends class with the exception of a few people who absolutely don't come to class. Perhaps people who left the course and the administration just did not remove them from roll calls? What about experience?

We do a regular point plot to see change in attendance over months. The vertical lines along the points denote a 95% confidence interval.

The third years have found that perfect balance of leisurely attendance during most of the semester and then picking it up during the last months. First years exhibit that trait of a "नया नया मौलवी" perfectly. Second years have figured out that attendance does not really matter, but are still to find out how badly those marks can affect their marks. With time comes wisdom,

That's it for now. I'd love to do some more if someone can come up with data at a lower granularity.

Good but not Great

People are usually good, and rarely great. Most of us live in a story where we have no defining purpose. A story following a path as simple as the thousand tendrils of a tree, taken root for a century or two. A purpose which so drives us, that a single word may define most of what we are made up of. Most people are good folk. They hear no evil, they see no evil and do no evil. They also hear no good, see no good and do no good. They are for all intents and purposes simply there to fill space and "exist". In the immortal words of wise men of the past,

There are those who change according to circumstances and those who do not. All progress must then be left to those who refuse to adapt, as they are those who impose their will on nature.

Some however strive to be outliers. They notice things and act on them. They are the ones who do not apologize for their opinions and those who do not fear their own actions. These are the great. The great however, are rarely also good. The good shy away from change as a touch-me-not from one's skin. When the great force it upon them, the great are painted in colours belonging to shades of evil. Rarely does a change yield immediate results and so rarely are the great also seen as good.

The good are just as important as the great in determining the world's fate. The difference is that the good determine it by their inaction and incompetency, and the great do it by their action. It takes monumental will or addiction to make a great out of a good.

Phone As Raspberry Pi Screen

I usually have to work on a cluster in the college and all I need is a client to connect to that cluster. The Raspberry Pi is a great alternative to my laptop which has recently started to die of old age.

The problem I face with the PI is however quiet a different one. I usually don't have a projector / display around to see what I'm typing in. This is a workaround to that problem. Since I almost always have my phone around, I decided to use that as a display. 

Do note that this does not mean that you will have full graphics. It just let's you see what you are typing on a terminal in the Pi. For full graphics you would need some sort of VNC setup.

Let's get started.

1. Hardware Required
1. Raspberry Pi
2. Wifi Dongle
3. Keyboard
4. Mobile Phone (Android / iPhone etc)
5. Working internet connection (For setup purposes)
2. Software required
1. On Raspberry pi
1. ssh-daemon (This is preinstalled if you are using NOOBS). Otherwise use sudo apt-get install sshd
2. Tmux ( can be installed via sudo apt-get install tmux)
3. In case you don't like tmux for some reason, you may be able to use screen in a similar fashion as described.
2. On phone
1. Some sort of SSH client. I use Juice SSH but you are free to use any one.
2. Since I have never used Apple, I'm going to leave finding an SSH client there to you. comment below if you find one that you have used.
3. Other requirements
1. A keyboard for the PI would be nice, though you can use your laptop too in case you do not have one.
2. The Raspberry pi needs to be able to connect with a network beforehand. See this resource.
3. In case you have a router lying around, connect the Pi, phone (and laptop if needed) to the same router. Otherwise set up a WiFi hotspot in your phone and connect everything to that.
4. Steps
1. Power on the Pi
1. Switch on the Pi.
2. Plug in the wireless dongle.
3. Let the Pi connect to the network
2. Power on the laptop
1. SSH into the raspberry pi
1. ssh pi@192.168.xxx.xxx
2. Windows users can use Putty
3. Install Tmux via sudo apt-get install tmux
4. use raspi-config command to make sure that the login procedure is login to command line
5. exit the ssh session (Ctrl + d)
3. Reboot the Pi
1. Connect a keyboard
2. You will be logged in to the PI user automatically as per the last config.
3. type tmux new-session -s main
1. This opens a new tmux session.
4. SSH into the Pi using your phone (Juice SSH)
5. Using your phone type in tmux attach -t main to the terminal you have just opened.

You can now see the same tmux session which is running on your PI. Whatever you type in the keyboard appears on your phone. This allows you to use your phone as a temporary display for the Raspberry pi terminal.

Here's a demo video.

A Regular Expression Engine in Python

Code is here

Regular expressions are quiet useful in programming tasks. They are a useful method to search for patterns in long strings of text. For example the pattern "(c|b)at" allows us to search for either 'cat' or 'bat' in any string that this pattern is run over. This example was a simple one and regular expressions in practice are far more powerful and complex.

One must take care however in remembering that regular expressions are a loaded mathematical term. They are not the same as regex and regexp since those two do not make regular languages.

Here, for the sake of brevity we shall call regular expressions regex. Regexes are usually made using Deterministic Finite Automatons. They take a string of input symbols and tell us if the DFA accepted the symbol string or not. Such an activity is called the run of the DFA.

Steps

1. We define that our regex engine will only provide the basic elements of concatenation, union and kleen star. Thus '|*' are the only special characters allowed besides the other things in the alphabet. 
2. We first build a Non-Deterministic Finite Automaton with epsilon transitions using Thompson's construction algorithm from the given pattern.
3. Then we convert this e-NFA to a regular NFA without epsilon transitions by the epsilon closure method.
4. The new NFA is converted to an equivalent DFA by the power set construction method.
5. The DFA is finally run on a string. The Union (|) and the Kleen closure (*). Concatenation is assumed for consecutive symbols.
6. Upon completing the run, the DFA either returns True or False depicting that it has accepted the string or rejected it.

Example

For the regex '0|10*' (which means either a zero or one followed by any number of zeros) on the alphabet '01' the following strings return results as shown:

• '1' : True
• '0' : True
• '01': False
• '10': True
• '11': False 

The entire code is available as a gist.

A Better Picture of Graduation

Just thought that since everyone goes about quoting graduation marks without realizing how misrepresented they are, here is my graduation with all the tests on record.

Practicals + Theory + Internals

Why You Should Code

Every day, every second now, we are immersed in a world where dumb machines work for us. To list the obvious, your phones, TV, desktop, watch and, laptop are all computers. In the not so obvious list come power plants, rockets, cars, water supply, GPS, and, the post office. Despite living in this obviously computer driven world very few people know how to code. So I'm going to tell you why I code, in an effort to get you to pick up yourself and get going.

Programming at it's heart is communication. You instruct the machine. You tell it what to do, your desires, the things you don't like, the things you want more of, the things you want done quickly. You tell the machine what it is that you want done. In those moments of history, you are God. You can instruct the machine to anything and it will obey. It will obey in a manner you never thought possible.

There will be no complaint, no groans. There will be no raised eyebrows. You have a powerful ally at your command. You can designate repetitive tasks to it and it will do them till the end of time itself. You can assign some intelligent tasks to it and it will do them till the end of time. What the machine does is up to you.

The first problem I solved was downloading YouTube videos. I knew I wanted this entire play list and I did not want to go on clicking anymore. So I wrote a program for it and the machine obediently downloaded the entire play list for me. That is all there is to computing. Nothing fancy, nothing mysterious. Simple communication.

I now solve a lot of problems using my laptop. I have to apply for a PhD. So I write a program to suggest names of PhD guides to me.

There are lots of ways to communicate with the computer. Sure you can use something like Excel to do calculations for you. But in that you are limited to the options that the makers of Excel provide to you. Imagine this like talking to a person with cards. You can only say what is on the cards. Of course you will be limited! On the other hand, the way for unlimited communication is programming languages. Pick any one!

All programming languages are designed with one thing in mind. How can we maximize the things you can communicate to a computer. With a programming language at hand, you can communicate with your computer in an almost unlimited fashion. So learn one.

Try out Python! Try out Matlab( or Octave if you prefer free software like I do)! Try Java or even C!

All these languages allow you to communicate with the computer. All of them are powerful! With an untiring assistant at your beck and call, your life will be a lot simpler.

Guidelines To Cheating In Exams.

Read on with a pinch of salt.

First off let us describe the examination. Here we consider a scenario where candidates sit in a hall and are given some sort of question paper and answer sheets upon which answers must be written.

There are one or more invigilators and they keep the cheating in check. Ok then let's get started.

On an individual level

• This is the ultimate gig. You are a one man army of complete incompetence and are set up to fail but are not prepared to give up on the test yet.
• I admire your spirit but WTF girl(man if you please), you should have studied.Ok, let's see what you can do.
• Before the exam:
• Remember that short term memory is pretty useful in such situations.
• Risk: low
• Use: high
• Volume: low
• Death: you start writing before the writing time.
• Memorize key information. If you do not know what is key ask around (remember that you are in a room full  of people who studied)
• You will forget all of this in due time (it's called short term memory for a reason) but you have about 15 minutes to jot it down on the answer sheet before that. 
• Chits are a classic way to cheat on an exam.
• Risk: low-medium
• Use: high
• Volume: medium-high (depends on writing size)
• Death: If you get caught by an invigilator (more on that later)
• You essentially write stuff down on a piece of paper.
• You hide it somewhere (washroom, your clothes etc)
• You see what is required.
• You dispose it (throw it from a window, eat it ?)
• Environment tampering
• Risk: high
• Use: medium
• Volume: low
• Death: If anyone sees it you are done for
• Involves writing things on walls, desks etc instead of paper as in the chit model.
• I would prefer a better model of cheating.
• During the Exam:
• You had time and you did nothing. If you are reading this you probably tried and failed. Ok, moving on.
• Unless answers come to your mind magically, you will have to ask someone the correct ones.
• I don't know. If you have a good one tell me.
• After the Exam:
• While the paper is in transit.
• Have a genius friend create a copy of answers for you.
• While the paper is in transit to the examination authority, switch the answer sheets in a Mission-Impossible style.
• Pray

With a group

• Now this is a different beast altogether. You have allies!
• Distracting the invigilator:
• The invigilator is human. Meaning he has the same model of attention that everyone else has. Meaning he can be putty in your hands.
• Help me!
• Decide on an initiator and reciprocator among the group.
• Initiator calls the invigilator to her side to clear up a 'misunderstanding' in the paper.
• Everyone else quickly does the dirty while the invigilator is occupied.
• Reciprocator then calls the invigilator to her side and the Initiator then completes her cheating.
• Something strange
• Drop your pencil box, get up suddenly from your seat, ask for extra sheets. In general do something which grabs attention.
• For the rest of the people, while the attention is diverted, exchange info. 
• Keep them walking
• Asking for more sheets is a good way to keep them walking and thus keeping a moving field of attention.
• An invigilator in one place is a dangerous person. they see everything and you do not know where they are seeing.
• Call signs:
• For MCQ type examinations, touching the nose for option 1, ears for option 2 etc are common things to do.
• I have known this to work across 2 different halls.
• Answer sheet circulation:
• Using any of the above methods of distraction, exchange answer sheets between groups.
• Use rules of exchange, thus making sure every one's chances of passing the test are maximized.
• For example, "Pass clockwise"

Those are the things which come to mind off hand. If you have more(I know I have missed a lot) add them in the comments below. Cheers!

Teacher Relative Marking In Schools

Machine learning has brought to my attention the horribly wrong interpretations possible due to data not being on the same scale. When information is not measured by the same stick, the result is that the bigger stick dominates the inferences you draw from the data.

In schooling, standardized testing is a method with good intentions in mind. It enforces the same standards on all the people taking the test. In the context of national education however, I think there might be something that the authorities have missed.

Not every teacher is a great one. I have been fortunate enough to have been taught by good teachers. I have also had the fate of being instructed by absolute teach-shop employees. since on a national scale we would like to measure the learning ability of a child, it would make sense to check if there is anything to be learnt at all where the child is studying.

I propose that along with the students, the teachers also take board exams and so on. Then when the students get their results, their performance is weighed in relation to how good their teacher is.

A student scoring 50/100 when his/her teacher scored 30/100 is a good student. They have learnt a lot. A student scoring 50/100 when the teacher scored 90/100 is not such a good learner.

This would be a more fair scale than every student getting the same paper irrespective of the environment they studied in.

I do realize that this could have potential disasters since the student's performance no longer depends on them only. It would put enormous pressure on the teachers as well since they would effectively keep on taking exams throughout their career.

One potential fallacy is that over time teachers might get exceedingly good at taking tests and thus fail the system altogether. They will learn the common questions and so on. Since subjects are finite in nature at the school level, this is bound to happen. It might result in the teachers getting better at their subjects over time though, I cannot predict this well enough.

To conclude, I say that on the day of the exam, let the teacher sit with the students ad face the music. Then let the students be judged relative to the teacher.