SparkDC - automated file sharing over ssh

SparkDC is a simple command-line a tool that automates file sharing and downloading over ssh among a connected set of users through a central server. This version is specifically developed for IIT Delhi network. Nevertheless, the same hack can be extrapolated to any other network that has a common root folder and that has a good upload and download speed. These kind of networks are normally academic LANs and corporate networks. The code is on github.

How It Works
Here's a short description of how all this works. Consider bunch of users who have started these scripts, in particular consider users D(Wants movie Inception) and S(Has movie Inception). The script writes all the file names S has chosen to share on the network to one Host folder. This happens for all the users. So we now have the information of which files each user has. User D can search through these file names through a script search.py. When D searches for say, Inception using search.py, it greps over all the file names that have been shared  and finds that S has Inception(Only their names are shared, a meta file). When he wants to download Inception, the script having found inception at user S, will create a request meta file in the User S's home folder. Since another script on S's machine is listening for requests, it will upload the Inception to D's home folder. So now user D has downloaded the file Inception to his home folder from user S's local machine.

Necessity is the mother of invention (little story)

In our campus we used to have LAN sharing through direct peer-to-peer protocols using standard softwares like Odc and DC++. But all of a sudden, administration decided to block these protocols due to excessive addiction of students to LAN gaming and watching movies. But they didn't block shell access to home spaces (around 8 GB for each student) on the cloud.. So it was not very much difficult to exploit these home spaces and automating file sharing and transfers on demand with Python. I have written these scripts around Oct 2011 within two days of the idea. The scripts were just able to do what they were meant to do. So, there is no graphical interface, there are no aliases and stuff. These are some minor glitches I didn't care to fix, as the LAN services are started again just on the day I was going to release this (yeah, exactly the same day :( FTW), and there isn't much audience hence. If at all the LAN is blocked again, I am going to work on it :D. For further details about the implementation you can contact me on Twitter or Facebook.

Here are the instructions on how to use the Spark specifically for IIT Delhi LAN network.

If you can figure things out on your own, read Short Instructions otherwise read Detailed Instructions.

Short & sweet Instructions

1. Download Spark and extract it to a spark folder on your computer
2. Set-up ssh keys for IIT D ssh server, if you haven't already.
3. cd into Spark directory. Start Spark by the command 'python start.py YOUR_ENTY_NO_HERE> FOLDER_YOU_WANT_TO_SHARE ', this program shares any files in the folder you have shared and listens to requests from other users and uploads automatically. Leave this running.
4. To start an interactive search and download prompt, open another terminal and run the command 'python get.py YOUR_ENTRY_NO FILE_NAME_YOU_WANT'

That's it! You are rocking with Spark. Below, I have attached some screenshots of Spark for you.

Detailed Instructions
Downloading the spark

Head over to my github repository. You can see links 'Download as zip' and 'Download as tar.gz' on that page. Download it and extract to a folder named spark anywhere. I haven't added any aliases (shortcut commands) to start the script. So spark has be started whenever you want by 'cd-ing' into this folder.

Initial set-up

1. Spark is bundle of Python scripts so you must have Python installed. Here you go for Python.
2. You must have your ssh-keys setup. Whoa, if this sounds scary, just head over to detailed instructions here. In case of our campus, server name would be something like ph1080614@ssh1.iitd.ernet.in.You can login to your proxy server without a password now! I suggest you not to add any pass-phrase, unless you have some hacker friend who messes up with your computer.
3. Now we have to do a little edit to the file start.py. The default HOST (for CS, EE, PH) on cloud is my folder. In future if the host changes you can edit in this file (line 16 of start.py). (This is like adding a new Hub address in ODC on our LAN). For other departments you may have to have a volunteer who can give a restricted access to their home spaces. Anyone interested further can contact me

Using Spark

That's all the setup to be done, when you first download spark. Now, here I explain how you can use spark to share and download files from different users sitting anywhere in the institute or any hostel (yeah, that's right, anywhere!).Of course, only catch is sharing is among only under same HOSTs.

To use Spark, there are just 2 simple steps. First, we have to start Spark.

1. Open a terminal and change to spark directory. And run following command

         'python start.py YOUR_ENTY_NO_HERE FOLDER_YOU_WANT_TO_SHARE '.

You can see how I started spark on my computer in the screen-shot below.

Just like when you start your ODC, and tell it which folders are to be shared, you have to tell Spark which folder you want to share.  Here I am sharing a folder called 'odc'. You can specify an absolute path, like '/home/sravan/odc' or relative path, like '~/odc'
If you have done any typing mistakes, spark is going to tell you what the mistakes are. Here, mine is successful and Spark is now running. This program (start.py) is listening to requests from other users who are using Spark, if someone wants any file you have shared in odc, the porgram automatically uploads to their folder. So, we have to keep this terminal open, much like you keep ODC open, even if you are not downloading or searching.

Now it's time to download some file! To search and download a file (let's say, Inception), we have to start get.py a script which searches for the files all the users shared and downloads it to your proxy folder.

2. Type 'python get.py YOUR_ENTRY_NO FILE_NAME_YOU_WANT' to start an interactive search and download dialogue like following.

This is an 'Interactive search program'. So it pretty much guides you through downloading the file you want to your proxy folder. The speed will be around 1MB/s. It will  be downloaded to your proxy folder though, not your computer. (I could have fixed this, but the ODC is restarted on the very day I was testing and finishing, in future if ODC is blocked again, we can add this feature, contact me for further details.)
Once the file is found and is being downloaded, Spark moves onto next search prompting you for a new file name. Sometimes, Spark may give you cryptic error messages, even if the file is  correctly being downloaded to your proxy folder, you can neglect those messages, after checking for it in your proxy folder.

That's all, now you can share files and download files from anywhere in the campus!

Further development

As I have told, the same scripts with very little change (couple of variables) would apply to any other network of shell users.This version has a great scope for further improvement in terms of user interface and platform independencies. A more general purpose graphical version of the software can also be made. If this sounds applicable to any network you know and fun to build, I urge you to contact me on Twitter or Facebook!

The whole universe is in a glass of wine.

It is priceless,to sit in your room quietly and flip through pages of your favorite book from past.
This is the one of the most humorous descriptions by Feynman of how we divide the nature into our own 'fields of study', and blissfully ignore everything else except our 'concentration'. It's an extract from his Lectures part 1.If you want to flip through it's page 38.

...A poet once said, "The whole universe is in a glass of wine." We will probably never know in what sense he meant that, for poets do not write to be understood.
But it is true that if we look at a glass of wine closely enough we see the entire universe. There are the things of physics: the twisting liquid which evaporates depending on the wind and weather, the reflections in the glass, and our imagination adds the atoms.
The glass is a distillation of the earth's rocks, and in its composition we see the secrets of the universe's age, and the evolution of stars.What strange array of chemicals are in the wine? How did they come to be?There are the ferments, the enzymes, the substrates, and the products. There in wine is found the great generalization: all life is fermentation. Nobody can discover the chemistry of wine without discovering, as did Louis Pasteur, the cause of much disease.
How vivid is the claret, pressing its existence into the consciousness that watches it! If our small minds, for some convenience, divide this glass of wine, this universe, into parts—physics, biology, geology, astronomy, psychology, and so on—remember that nature does not know it! So let us put it all back together, not forgetting ultimately what it is for. Let it give us one more final pleasure: drink it and forget it all!

Creative Writing in the Exam Hall !!

Who is excited about creative writing?!

Well, don't be afraid, I am gonna introduce you to a very different kind of 'creative writing' that I recently found on the web. This kind of writing is called 'Newspaper Blackout Poems'.Have a look at some of those on the following site

................................................. NewspaperBlackout Poems by Austin Kleon .........................................

If you haven't understood from it, It's like writing a poem but......

Instead of starting with a blank page, as a Blackout Poet, You grab a Newspaper and a permanent marker and eliminate the words you don’t need. There! You have it now(umm. may be not)

Here's one such piece I have written myself. Take a look

(Hover the mouse on the image to zoom in, off the image to zoom out, IE doesn't support this Try Chrome or Firefox)

Yup, its not a newspaper. Its my major exam question paper. You can call it 'Question Paper Blackout Poem'.Funny eh?! I have made that in the Major exam.Apparently my creative quotient peaks in the Exam Hall.I thought its very easy to create writings like that when I read it in the website, but believe me it is not so easy. Try with any printed paper yourself.

You must argue how can I black individual letters out of the whole words, but how the heck am I going to get a 'FUCK' in an exam paper.

Happy Blacking out :-)

tECHO

Let me get little geekier this time
If you are the kind ' what's the big deal with IE9 or Chrome or safari', u better stop reading this. This is supposed to be a tale of frustration regarding the browser giants.So, leave this post if you are that kind, and haven't left yet :).


So, there was the announcement of IE9, how do i get to know all this small 'uselss' news!?You can't kill time at IIT.You always have time left-out.Little interest, is all that required! I was seriously looking ahead for some VOIP providers.........let me tell all this useless story about why the interest over IE9' after it's.., u can call it review.


I hate that restart after installation
First, installing the IE9. This is often overlooked whenever a software is reviewd (best call it an App). I tried more than 5 times to install the IE9 Beta before its release as tech enthusiast. All unsuccesful. May be I don't know how to keep my vista clean, which is why the installation problem.Let me not blame IE9 for this.So finally it's released.I downlaoded the tiny installer, not the usual 'exe', which is the only available one for vista.It goes on to downlaod further installation files without telling me the size or approximate time. This is a serious issue, for folks who have limited usage, or limited speed. It took some decent time to finish the installtion.



Did i say 'finish'. No, not yet.IE9 is product of Microsoft, and here I am trying to install it on it's native windows OS, damn, and it asks me for a re-boot !!Look at chrome, even if you install 10's of apps n extensions, it doesn't even ask for a browser restart, such smooth installtions, who doesn't like it! And here is IE9, itself the family of microsoft asks a complete system reboot.Well, embracing latest trends in digital tech is what makes us geeky, not just 'Zuck' level coding.


Damn Yeah.....'Your websites shine on IE9'
So, in spite of the disappointment I went on to close all my activities on the comp to enjoy IE9 experience, and clicked to reboot. Damn again, its making those 'Configuring updates XX% complete' thing. And I waited like an apple fan in the 1-mile que to buy that iThing, for it to restart! I was really eager to try IE9, partly because, the most ethical company in the world, Microsoft, is so dedicated to push the web forward by setting up a special site to kill its own previous IE's. And also I found on the web that MS took a flying leap in IE9's improvements.
So, finally the installation is complete.Click! Here it is, unlike a modern Chrome browser it takes a stunningly longer time for initial loading, far better than previous IEs though :). Wow!! The websites 'truly shine' in latest IE9.Thanx to such larger vertical size. Tabs truly revolutioned! Tabs are picked off, from down-address position to the right of it, making wider space than ever.And also they look elegant.You won't even feel the existanece of browser.The websites are Stand-Out entities in IE9, 'Your websites do shine in IE9"

Omni-box finally in IE9 too
One more major step-up in IE is the address bar+search like the omni-box in Chrome.But this unified address and search box became so standard in modern browsers, that you may completly overlook this issue, if you are an avid Chrome user, habituated to omni box.
Finally,you can feel the smoothness of Chrome in IE too, be it in flipping tabs, navigating. No no...not as smooth as in chrome. I just checked it again.Try this.Open some 5-6 tabs(can be empty) in IE and in chrome. Hit Ctrl-tab, little slower, to shift between the tabs. Felt the difference in smoothness from Chrome!!?In IE9 it kind of shakes up when you flip tabs, whereas in Chrome its seemless.

Nothing is perfect...so here !one problem
One usability problem now. Suppose you have some 5 tabs opened and you are on 2nd tab now,reading something, and you need to search something related to the one you are reading, you will open up the sixth tab n get the info.You wanna go back to the previous tab you were on.But,woolaa! u are lost now.The tabs are so tiny, as the address-bar n tabs-bar are sitting on same height, and look so identical that its not an easy step backward!Though this issue is the classic browser issue in other browsers too, the single line for search-box n tabs elevates this issue further in IE9. Color coded tabs come into rescue. If the new tab originates from the one you are in they will be of same color.A very good usability feature.

Add-on's!! Nobody knows more than the Google
Next, the add-on's or the extensions as you call it in chrome and safari. Either Firefox and Chrome truly excel at this and way ahead even Safari too, or the IE9 is too far behind from the standard set by Chrome, even with the latest update.There are two issues regarding add-on's. One is how easy you can find the add-on you are looking for, and the other, how easy does it get installed, develoeper population is not an issue as the skills on three browsers is entirely not different.

This is exactly where my frustration started. IE9 literally sucks at both of these. One example suffices to establish the point.I went to ie-add-on site, and typed the ubiquitous 'Ad Block' add-on in Bing's search box residing on the site. I am shocked to see 'No results found'. (http://bit.ly/f1iTdS)The site did look full of add-on's.I don't know if i have to blame the aspiring BING search or the IE9 add-on repository for being not able to find. So, my freind, it's not an easy task to find what you are looking for here.


Now the second issue, with semi-broken heart I browsed the gallery of add-on's to finally find a useful add-on, only to further discover that i need to download the add-on pack, manually and have it installed. Crazy....i didn't proceed folks, IITD sucks! hell of bandwidth limits!! So, the installation issue too sucks in IE9.

Chrome Rocks !!
Contrary to this, Google excels in both of these issues with it's Chrome. Try the same search in Chrome's extension gallery to find more than 150 meaningful results(http://bit.ly/f6KFxC)!! And the installtion is instantanous and seemless, as you don't need to restart the browser unlike in Mozilla.

There is no trace of imrpovement with the regard to 'internet options'.Its again the same old rigid pop-up as in previous IEs, with no added usabilities.Whereas the latest addition to Chrome, the unified 'preferences' or 'internet options' option in chrome, that opens up in an extra tab is revolutionary, no need of digital literacy, type in the options you want to tweak in the search-box, wohla there it is, it lists all the occurances in preferences of the word!! Revolutionary it is.


I had been saying to myself, no Bing doesn't suck, neither does MS suck, but literally Microsoft sucks.And this time a real Good-Bye to Microsoft!

A Poem ...kind of

I saw you from here, my Heart skipped a beat, your smile...
..........................................................................always does that.
You said this is unreal, It forgot the light-speed, your anger...
...........................................................................makes It tensed.
Did you look at me...?, It echoes YOU, your eyes...
...........................................................................told It something.


See....!!! I never thought I can get poetic.
(may be that doesn't sound beautiful, at least it should look beautiful. So here it is.)

huh...seems neither does this look good. this should http://docs.com/AOI8

Time for a new post!!

Its just a comment I made on an article in NYTimes. This post intends to drive your interest on that very particular issue, 'agriculture in India'.
The following is the link to the article.

"Article on NYTimes Galloping Growth, and Hunger in India"

Read the above article first and have a look at the comment below after that.

I really admire the true-consensus that NYTimes makes on very remote issues like the one about the farmer in this article.

My father is a farmer living in a small village in Andhra Pradesh, India.My dad has striven hard to send me to a school and provide good education so that i wouldn't have to work hard in the fields like him. And now I am in IIT, the prestigious institute.And I am going to get a big pay-check after my graduation (it's not a big deal for IITian :P). <<----- Yeah!! Come on, I am not getting that pay-check, but many get it, isn't it?!------>>

My dad has recently sold out the 5-acre farming land, as each year we are not able to even recover the money we put in the farming.There are many farmers like my dad with a good educated child, who are moving out of villages leaving the land.(Vastly i think this is the main reason behind increasing urbanization of India, well even the urban areas won't be able to sustain the increasing population)

There are so many other issues the article hasn't covered( of course, i don't expect such a large problems in agriculture system of India to be listed out in this tiny article).The point is, there is no modernization of agriculture, not at all, except for the one who has more acres of land and who can spend some money on the machine. My mom also started going to field as we are not able to recover the costs for machine and labor.And we have never stored out rice-grains in warehouses since 20 years i knew, it was always our own house.

There is an intense need to attract the youth to areas like these in India. There is lot of entrepreneurial-wind breezing in the Indian universities and there are lot of capable technological centers in India to modernize the agriculture.

At last, no individual, in gross, is driven by the dream to make India shine in world platform. Its his own belly he is trying to fill.So,I think if agriculture can be shown as a profitable modern-entrepreneurial venture, the elite of Indian students will be attracted.

Hope, one day, I will again grow crops in my sold-out land !! :D